Surface drilling Drupal Panels 3

This article contains some notes from a short reverse-engineering session targeted at Panels 6.x-3.8 and Chaos Tools 6.x-1.8. The goal was to capture some first impressions and record them for potential future exploration and understanding of that module. The motivation came from the current lack of technical documentation from the powers that be.

Before deploying the drills, I wish to share with you a few contemplative remarks about Drupal documentation. I feel disappointed by the supposedly "user-friendly" and sometimes overenthusiastic case-oriented explanations so prevalent in descriptions of Drupal ("if you want X, click on Y... how cool!"). Frankly, I find them much inferior to what I like to call model-oriented explanations ("the system consists of parts X, which have state Y and interact according to rules Z") based on strict terminology and aiming to identify intermediate causes at work. A good model capitalizes on your prior software engineering and site building experience by allowing you to generate tons of individual cases in a reasonable amount of time. On the other hand, a set of cases helps you solve only a limited number of problems quickly by imitation. A large database of cases takes much effort to collect, maintain and remember, and it does a poor job at eliminating residual risk. When you're hit with a problem not yet documented in the database and forced to somehow augment your existing cases, this task is going to take an extraordinary amount of time or require to call in heavy artillery (a consultant). In contrast, a well-constructed model takes just a little effort to acquire (but not to create!) and promises much more substantial risk reductions; model refinements do not require quantum leaps of imagination. By abstracting well, you can keep most issues out of your mind most of the time, and with a good choice of metaphors, related pieces of a solution should come together quite effortlessly. In effect, adopting a cohesive model-oriented approach allows you to deliver more consistent problem solving for your clients. Similarly, when you hire someone, they better bring this secret weapon in their arsenal rather than just Google + d.o.

Back to the main topic. A panel is a named, persistent set of specifications that govern a joint presentation of multiple pieces of content on the same page. These specs cover

  • which kinds of content or even which individual pieces of content should be retrieved from the database;
  • where (in which pane of a multi-pane layout) each piece of the retrieved content should appear;
  • how each piece of content should be rendered;
  • under what circumstances and for whom each piece of content should (dis)appear;
  • (for panel pages) which URI paths should trigger displaying the specified content.

A panel is defined by a site builder. It is stored either in the database (by default) or in code (when exported, e.g. for the sake of VCS). It is then used by Drupal as a blueprint for page construction whenever its associated path is requested by the browser. The specifications that comprise a panel are further subdivided into a (non-empty) set of variants, each of which may differ from the other variants in certain respects while sharing the common base configuration to foster reuse. (However, variants are unlike displays from Views in that they all have the same rank; there is no default variant that others would inherit from and override.)

Panels are somewhat at odds with Drupal core, meaning that they seek to replace rather than just expand upon and integrate with some of the previous ideas:

  • In their capability to lay out pieces of content, panels build upon—but also to an extent compete with—page templates defined by a theme. In particular, panes of a panel layout are conceptually equivalent to regions of a page template. Because a panel is placed within a region, it can be also thought of as a means for subdividing the region into multiple sub-regions (which are not known to the theme subsystem).
  • In their capability to retrieve and render pieces of content, Panels developers abandon Drupal's core idea that all content should be represented as either nodes (for user-defined content) or blocks (for reusable pieces of UI). Other kinds of HTML content, not belonging to either category and possibly computed on-the-fly, can also be made available to and rendered in a panel.

To put the above and following explanations in context, consider the following screenshot of the Panels UI (with some section numbering added):

Panels UI

The configuration sections are subdivided into three main areas: Summary, Settings, and Variants. The information under Summary and Settings concerns the panel as a whole, while the subsections under Variants pertain to each of the defined named variants. With a few exceptions, the section labels are not self-explanatory at first glance, and the information contained within each section cannot be easily understood separately from the other sections. In this sense, the UI is non-intuitive. It won't teach what can be done with panels in a learning-by-doing approach. It will cause concern with regard to unexplained settings and overly abstract terminology. The module author's line of defense is that you don't have to understand every piece of the UI to successfully use panels for simple layout scenarios. While true, the suggested adopt-now-explore-later approach also conveys an ominous message: if the documentation is not ever written—and if the adoption or future development of the module suffers—you may find yourself in a bad position by having placed bets on it. (It is not much consolation for new users that the friendly competition doesn't fare much better with regard to making their alternative comprehensible and that the Panels module is well-entrenched with a significant user base.) The question is: can we in the meanwhile remedy this situation by sacrificing some time on the altar of open source and simply peeking into Panels code? Or is the code as poorly done as the documentation? The rest of the article is my first shot at answers (for the impatient: yes and no, respectively). I start by exploring some of the more confusing—while at the same time important—sections of the Panels UI and conclude with a brief foray into code design and implementation issues.

It turns out that the key to understanding Panels configuration are the concepts of "contexts" and "content". These technical terms are not at all related to the Context module (best erase it out of your mind while studying Panels), only weakly related to the content types of the CCK module, and they might not match any colloquial meaning taken from a dictionary. Such is the nature of terms unleashed on the unsuspecting world at night by well-meaning software developers. Let's focus on "contexts" first. In Panels terminology, the term context refers to a specification of a piece of (renderable) content that should be loaded and become known under a chosen id within the panel variant before it is output as HTML. For example, if your panel should display three nodes next to each other, it would obviously first have to load them, which you would specify by adding three contexts of type node to some panel variant. The Contexts section (2) lets you directly specify the pieces of content that should be loaded; for example specify a node by its nid or title. Alternatively, you could parametrize the panel to load different pieces of content depending on what has been provided through the URL. To achieve this, you would include placeholders in the panel's path (as shown on the screenshot) and then use the Arguments section (1) to associate each placeholder with a context type (e.g. specify that the first placeholder called %node1 really refers to a node ID rather than a taxonomy term ID; at request handling time, the URL is first taken apart by Drupal's menu subsystem and for each placeholder all modules, including Panels, are consulted – see _menu_load_objects in includes/ for details). Finally, by using the Relationships area of the Contexts section (2) you could also specify a context that loads a piece of content related to another, previously specified, piece of content (say, load the user who is an author of the second node loaded by the node context). Based on the preceding description, a better term for "contexts" might have been "content loaders"; alas, the release night is over, and it is too late for changes now.

Now let's turn to content. In Panels terminology it refers to "something that can be placed (and somehow rendered in) a pane" through the Content section (3). Unsurprisingly, the pieces of content loaded by contexts as well as their individual parts (such as CCK fields of nodes; or the user picture from a profile) fall into this category. For this reason you are generally well-advised to first define contexts before even turning to the Content section. However, other pieces of content (such as blocks or HTML snippets) are also available without the need (or possibility) to obtain them through a context. The advantage of loading panel content through contexts is that the attributes of the loaded objects then become available within all panes as variables that can be used in string substitutions (e.g. %node1:title or %user1:name, where the prefix equals context id). For example, having established the three node contexts and one user context as mentioned above, you could add a HTML snippet (aka "custom content") with the node titles and the user name to a pane.

Having acquired a basic understanding of the context/content pair of concepts and experimented a bit with the Panels UI, here are the next few natural followup questions that come to mind:

  • Where do all the selectable context and content types in the Panels UI (and their configuration forms) come from?
  • How does the Panels module know to offer "Node", "Taxonomy term" and "User" contexts and how do you know what data exactly they load?
  • What the heck is a "Node add form" context supposed to load and how is it different from "Node"?
  • What determines which items appear on the "Miscellaneous" tab when adding content as opposed to the "Widgets" tab?
  • What if you want to add something that is not at all listed in Context or Content sections?
  • How exactly is the Relationships combo populated?
  • Should we degrade ourselves to click monkeys and figure out all of the above by trial and error?

I won't even come close to providing detailed (boring?) answers to all of the above questions and will instead address the last (somewhat rhetorical) one by providing a little push in the right direction. As announced before, the solution is sought (and found) by examining the code. It turns out that most of the items and forms you come across while interacting with the Panels UI are not hard-coded anywhere in the Panels module, but rather contributed dynamically by code. "I know – panel hooks!" might be your first guess, but unfortunately it's not that simple. There is a lot more indirection in the Panels code than meets the eye and a more refined mechanism than good old Drupal hooks is at play. Furthermore, it turns out that neither contexts nor content types are designed to be panel-specific. In fact, most of them are not even implemented in the Panels module, but rather belong to the Chaos Tools module, upon which the Panels module depends. In general it's safe to say that the indirection and abstraction sauce is good for flexibility and extensibility and also a definitive poison for code comprehension. The case of Panels is no exception, and so I guess that the author might be playing with fire to some extent ("we have this wonderfully flexible super-duper API for every conceivable task, which unfortunately noone besides us can wrap their head around). I'm glad to report that I have seen much worse and that the code design and inline documentation are definitely more solid than what you see in "advanced help" today. In short, you can wrap your head around it.

To navigate around Panels code—and to see how the administrative UI is built and what each selectable item stands for—you have to first grasp the much more general concept of CTools Plugins (reified hooks). This part of the Chaos Tools module is an example of the Strategy pattern at work. It aims to facilitate a division of labor between module developers and site builders. A plugin instance is an exchangeable persistent object responsible for performing a set of tasks in its own particular way. Plugins are typically selectable (and so instantiated) through the UI of those Drupal modules that depend upon them (like Panels or Views). They may also contain callbacks that construct an administrative form-based UI to allow site builders detailed configuration of each plugin instance. Alternatively, plugins may be organized hierarchically (in a type:subtype or parent:child fashion), for the benefit of a more structured selection UI. The exact set of responsibilities for a plugin type is usually specified by the module developer who invokes plugins of that type from her module. Particular implementations of those responsibilities are then provided as plugins by (potentially) different developers. A plugin is usually implemented in a single .inc file. Plugin names must be unique within a plugin type, but do not need to be unique across types. Plugins are used by modules or by each other. The CTools library acts as an intermediary and a directory service for publishing, locating and invoking plugins. This terminology and the discussed relationships are illustrated by the following examples:

Plugin type Responsibility of plugin type Plugin Plugin functionality Plugin implemented in Plugin used by Plugin instance (examples)
contexts Provide renderable content for a panel. node Load and provide data from a node specified by nid or title. ctools/plugins/contexts/ panels.module (indirectly, through its own plugin panels_context of type task_handlers) node1 and node2 in Variant1 of panelpage1
" " user Load and provide data from a user profile by uid or name. ctools/plugins/contexts/ panels.module (indirectly, through its own plugin panels_context of type task_handlers) user1 in Variant1 of panelpage1
content_types Render available content within an individual panel pane. node Render available node content. ctools/plugins/content_types/ panels.module (indirectly, through its own plugin standard of type display_renderers) "Node 1: ID" content in left pane and "Node 2: ID" content in right pane of Variant1 of panelpage1
" " user Render a picture from user profile. ctools/plugins/content_types/user_context/ panels.module (indirectly, through its own plugin standard of type display_renderers) "Node1 author" user picture in Variant1 of panelpage1

The required naming conventions and steps for defining plugins (and plugin types) and invoking plugin instances are documented in "advanced help" shipped with the CTools module, which you are advised to read carefully. The CTools library is used pervasively and consistently throughout Earl Miles's (aka merlinofchaos) modules (and also in some others). Thus, if we view Drupal as a domain specific language, CTools should count as a potentially prominent dialect. Becoming familiar with it will not only help you understand Panels, but also Views and potentially future of Drupal core, if it should spread into more mainstream use.

Note: while looking at the CTools module, I also found that it supports third-party module developers in evolving and versioning their own modules' APIs, including catering to different client modules using different versions of an API at the same time. The term API here refers to the set of hooks which the developed module invokes (i.e. the hooks provided [not the hooks implemented or library functions offered!] by that module). When CTools versioning support is utilized by a module, hook implementors must partition their code into multiple files, each of which corresponds to a particular version of the API. When the hook provider is about to invoke a (particular version of) a hook, it can query CTools for all matching implementations, while excluding any non-matching (e.g. too old or too new) implementations. This mechanism, documented in "e;advanced help", appears to be almost unused, and probably will remain so due to its confusing official description. Nevertheless, it is interesting to see an idea of this sort in place, given Drupal's track record of neglect for backwards-capability (at least between major versions).


Matthias Vandermaesen said...

Brilliant write-up! Read it with much interest. Indeed, Panels and CTools are a niche on their won with their own terminology within the Drupal ecosystem. While I used Panels in projects before, gaining a good understanding of what contexts, content, variants,... are, proved to be rather hard lacking unambiguous documentation.

jpl said...

@Matthias: Glad that you liked it. I ran out of time while writing this one, so it's actually not as well-rounded as it ought to be. Certainly Panels deserve a deeper treatment.

xkater said...

yes - this is great . . . and i love your 'tongue-in-cheek' ironic wit regarding, as it often seems, the programmer's incessant internal argument - program vs explication - the continuing requirement to make the 'implicit explicit'!

FitzChivalry said...
This comment has been removed by the author.
FitzChivalry said...

Thank you for writing this up from a programmer's perspective! I have been confounded trying to use many of Drupal's features and plug-ins because they're hard to grok. Your writeup definitely made Panels easier for me to understand!

Butler said...

Thank you so much Mr. Ploski - since I have been floundering around w/ Panels and am not sure I am up to more than click-monkey-ism finding this made my day. Your comments about model-driven vs example-driven documentation were so right-on. The tendency to provide example-driven documentation is something about Drupal that has often driven me to distraction, and it is not usually easy, and sometimes perhaps not possible (for me) to understand what goes on under the hood.

Thank you again.

jpl said...

You are welcome. Actually, reverse-engineering explanations/intent isn't very difficult with Drupal (it's pretty self-contained, single-threaded, easy to instrument/debug, and there are plenty of examples from which to generalize). You just have to ask the right questions (and pump in the right amount of time, alas).

Anonymous said...

bullshit to a non-programmer - isnt Drupal just another program for geeks and not what its claiming.

jpl said...

@Anonymous: This article is very much meant for programmers.

I guess Drupal indeed does have a problem of catering to different audiences, which sometimes makes both of them unsatisfied. Non-programmers are put off by implementation details appearing all over the place (in GUI), while programmers are put off by a lack of high-quality technical documentation, which forces them into inefficient styles of work (through GUI).

Personally, I view Drupal as a framework that should be used by programmers to build web sites that can be easily customized by non-programmers. To make it possible they have to hide much of Drupal's own "powerful and confusing" administrative functionality and unfortunate terminology. I think it's a wrong approach to give it all to non-programmers and hope for the best, but hey, it even seems to work for some people and scenarios.

Anonymous said...

These open source developers with their big words, leftist ideology, and utter contempt for anything resembling usable documentation are putting decent, hard-working Microsoft employees out of a job and leaving the rest of us to suffer the cost in hours spent spinning in lost circles.

jpl said...

@Anonymous As long as you can sell high quality "added value" projects/applications developed on top of that poorly documented code created for free by leftist open source guys (some of them being paid for it by definitely non-leftist VCs), you should be fine. Admittedly, that might prove difficult if you're just a hard-working Microsoft employee.

Imagine a world in which every software product would be just perfected to do one particular task well instead of being replaced, broken and reinvented ad infinitum. Now *that* would be dangerous for employment statistics. Fortunately, both Microsoft and the leftist OS guys seem to agree on the one point that spinning in circles is the bulletproof way to keep the software industry flourishing.

Anonymous said...

Thank you for taking my somewhat tongue-in-cheek comment somewhat seriously (I think).

In truth it is not possible for me to begrudge anyone who makes their work available to the whole world free of charge. And Drupal is indeed powerful.

It is however definitely possible for me to rip my hair clean out of my scalp in frustration sometimes nevertheless.

Your above article touching on such frustration is extremely well-written and very much appreciated.

John said...

Simple question -- how do I pass a %node:nid through panels, into a modules HOOK_block_view

Anonymous said...

thanks for the write on this. I am a hard core php guy and the terminology between drupal and panels and all the other stuff within drupal is very much lacking. It's almost like everyone is making up their own stuff and expecting everyone to just read their minds as to what that term means. Could I write all this in php, probably. Could I understand what the heck most of these terms mean within the framework of drupal... not so much. Thanks again. Jer

Anonymous said...

First, I appreciate the article, and too bad you didn't have time to finish it, since there is so much more about the system that needs to be explained, though i did find this helpful. Of course, I understand why you didn't have time to finish it -- because with Drupal, sometimes you have to devote a lot of time just to make the simplest things happen, and as your article shows, that deeper understanding sometimes only comes from examining the code -- but often the code only makes sense after you spend a lot of time as a "click monkey" trying things out by trial and error through the GUI so that when you look at the code you can recognize what it is doing on the other end. Just glad to know I'm not the only one who pulls his hair out trying to use Drupal. I've noticed that hardcore Drupalistas all tend to agree with the criticisms of Drupal, from the poor documentation to the obscure and obtuse terminology (that all Drupal developers seem to love to spew) to the unintuitive GUI's, but then defend it be saying "but it's powerful!" I'm just not sure that power has to come at the expense of sanity and clarity....

Earl Fong said...

In Drupal-land I've done some custom module development, but now need to learn more about site-building, e.g. using Panels.

This explanation / exploration is really invaluable and much appreciated for helping to actually understand some of what is going on and what the terminology means.

Reverse engineering someone else's software to create an accurate model, and then writing about it, is a lot of work. It's hard -- so thank you!

Post a Comment