plosquare.com blog

On errors masked and revealed

2011-09-11T20:20:00.009+02:00

In a recent talk professor Daniel Kahneman looks back at his research in cognitive psychology. In a side remark he notes that errors are "much more diagnostic about the mechanisms, so there might be different mechanisms for doing the same thing perfectly, and you're not going to be able to distinguish the mechanism by observing it". The approach of focusing on errors (or anomalies) in trying to accurately model something is of course a long-standing one. For example, we have the study of anomalies in medicine; natural experiments in social sciences; and of course the practice of intentionally breaking and perturbing things in about any other scientific discipline, including software engineering.

However, one of the striking characteristics of software engineers (perhaps shared with other kinds of designers, too) is that, unlike those scientists who attempt to peek under the hood of nature's work, we are obsessed with the idea of hiding the "implementation details" of our own creations - that is, abstracting away the mechanisms by which they operate to the greatest possible extent at all conceivable levels. I think that the driving force behind this is not so much the traditionally touted flexibility of exchanging implementations (which, to be honest, seldom needs to be done) and interoperability. Rather, it is our desire to simplify our own mental models, to reduce the cognitive load of having to think about (and worse, test) too many elements and interactions at once. Maybe so much so as to shift some of the processing from System 2 to System 1, in Kahneman's terms. We strive to reduce the probability of mistakes - which on the economic level also means reducing development and maintenance costs for our clients.

When software systems fail, the tables may be turned on us - the hidden mechanisms then become apparent, and the usually blessed lack of familiarity with the implementation details becomes a curse. We are then forced to learn things about our systems that we have never bothered to understand. In more extreme cases, this aspect may even represent a significant liability and risk to the maintainer, if contractual obligations demand quick, correct reactions.

I think a few conclusions may be drawn from the above:

Less intricate, "uglier" designs that reveal more of the underlying mechanisms, contain fewer layers of indirection, and put a smaller demands on supporting technology and tools might be actually preferable for economic reasons - more attractive from the point of view of contingency planning (if a total or maximum time-to-repair is a concern).
The skills of a software developer and those of a maintainer or troubleshooter are intricately linked, but nevertheless quite distinct. Great software developers know which internal aspects of (their) software to hide and which to reveal to make its normal operations easy to grasp and the rare breakdown possible to handle. Great troubleshooters know how to reveal those aspects that were intentionally hidden by the (possibly not-so-great) developers in order to understand the causal connections and required interventions. Ideally, they also have an extensive knowledge of the under-the-hood mechanisms of a particular software product or system which they maintain.
The skill set of a software developer is rightfully viewed as a superset of (and thus more valuable than) that of a troubleshooter or system administrator: a developer typically not only has to troubleshoot an analyze other developers' software because of the interfacing and teamwork requirements, but he also has to anticipate and support the future troubleshooting scenarios that will occur with respect to his own software. On the other hand, he can usually avoid accumulating truly significant amounts of factual knowledge about particular products or operating environments and rely on more general procedural knowledge.

I don't know about others, but I do seem to spend most of my software development time thinking about errors and mistakes - real and possible, static, dynamic, happening inside of my code and in the users' world around it.

Fix for "No such file or directory" while trying to mount --bind

2011-07-01T18:29:00.004+02:00

Problem: The command mount --bind /mnt/data/usr /usr in Linux fails with "No such file directory". /mnt/data is a mount point of a dm-crypt device. strace shows ENOENT for the mount syscall, no other clues. Variants that don't refer to /usr (for example, to /usr2 instead), do work as expected. The same error occurs after and during boot process, so it's not explainable by used files.

Solution: The problem disappeared after upgrading the kernel from 2.6.34 to 2.6.38.8.

Fix for random frequent freezes in GTK applications (Debian 'testing')

2011-06-26T01:42:00.006+02:00

Sometimes software configuration problems are solved thanks to a deep insight into how things work and methodical experimentation. At other times they are solved through a shotgun approach in which hunches, sheer luck and brute force are combined to get rid of a problem without ever truly understanding its cause. This leaves no useful information behind for avoiding the problem in the future, but maybe we don't care. The following describes such a situation.

After upgrading some packages, notably xorg, in Debian ('testing'), most GTK applications (e.g. Firefox, SeaMonkey, Eclipse, ...), started crashing intermittently, with just the infamous unresponsive UI symptom (aka freeze, hang). After some Googling I noticed that adding the --sync command line option alleviated the problem, and for a period of time I just used this as a crude but effective workaround. I got motivated to look for a real fix when I ran into the same symptoms while trying to run a hosted Eclipse workbench (a PDE JUnit test suite). Here the hosted workbench would freeze almost immediately after startup (the window rendered, but without the menu bar, after which no more repaint events were processed). It would happen always without the --sync option, but also very often despite it being supplied. I observed in the JDT debugger that the lock-up would always occur when the main thread was callnig a GTK/GDK function - not always the same function (sometimes gdk_flush, sometimes gdk_pixbuf_render_to_drawable, sometimes the main event loop). Stepping through the code would cause the problem to disappear (tell-tale signs of a timing bug; those are always the best ones). To make it even more "interesting", sometimes Eclipse would crash altogether with a "BadLength (poly request too large..." error from Xlib. Intuitive remedies, such as upgrading GTK/GDK packages or xorg or xlib, have not been successful, but this idea turned out to be (almost) right.

The breakthrough came after running ldd libswt-pi-gtk-3735.so (this is the SWT library used by Eclipse, which in turn depends on GTK), next determining which package each of the prerequisite libraries shown by ldd belonged to, and successively upgrading each of them. I didn't take care to repeat the test after each upgrade, but here is the list of packages that I replaced or tried to replace - after which the problem disappeared:

alsa-oss
binutils
gdk-imlib11
libatk1.0-0
libc-dev-bin
libc0.1-dev
libcairo2
libcairo2-dev
libfontconfig1
libfreetype6
libgdk-pixbuf-dev
libgdk-pixbuf-gnome2
libgdk-pixbuf2
libgdk-pixbuf2.0-0
libgdk-pixbuf2.0-dev
libglib2.0-0
libgtk2.0-0
libgtk2.0-common
libnspr4-0d
libpango1.0-0
libpango1.0-common
libpango1.0-dev
libpth20
libpthread-stubs0
libpthread-stubs0-dev
libx11-6
libx1106
libxcb1
libxcomposite1
libxdamage1
libxext6
libxfixes3
libxi6
libxrender1
libxtst6
xorg

Fix for ClassNotFoundException in PDE JUnit Test Runner

2011-06-25T17:15:00.004+02:00

When trying to run a PDE JUnit test suite in Eclipse, the requested unit test is not executed, the hosted workbench terminates immediately with the following error to the console (where org.epic.perleditor.editors.TestEditorAssociation is the name of the unit test). The plug-in which contains the allegedly missing class is present in the workspace. (This error also falls into the "it used to work before" category.)

Class not found org.epic.perleditor.editors.TestEditorAssociation
java.lang.ClassNotFoundException: org.epic.perleditor.editors.TestEditorAssociation
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClassInternal(BundleLoader.java:481)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:397)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:385)
 at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:87)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.loadClass(BundleLoader.java:313)
 at org.eclipse.osgi.framework.internal.core.BundleHost.loadClass(BundleHost.java:227)
 at org.eclipse.osgi.framework.internal.core.AbstractBundle.loadClass(AbstractBundle.java:1274)
 at org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner$BundleClassLoader.findClass(RemotePluginTestRunner.java:38)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClass(RemoteTestRunner.java:683)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClasses(RemoteTestRunner.java:425)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:445)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
 at org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner.main(RemotePluginTestRunner.java:62)
 at org.eclipse.pde.internal.junit.runtime.UITestApplication$1.run(UITestApplication.java:114)
 at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
 at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:133)
 at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3378)
 at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3036)
 at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2382)
 at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2346)
 at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2198)
 at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:493)
 at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:288)
 at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:488)
 at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
 at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113)
 at org.eclipse.pde.internal.junit.runtime.UITestApplication.start(UITestApplication.java:46)
 at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
 at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
 at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
 at org.eclipse.equinox.launcher.Main.main(Main.java:1212)

Solution: edit META-INF/MANIFEST.MF of the plug-in containing the not found class and add the class folder (i.e. bin) to the Bundle-ClassPath: directive, like so:

Bundle-ClassPath: cglib-full-2.0.2.jar,
 easymock.jar,
 bin/

If you need to debug this problem in more detail, the method responsible for finding the class (which works in a good configuration, and fails otherwise) is org.eclipse.osgi.framework.internal.core.BundleLoader.findLocalClass.

Fix for "Cannot complete the install because of a conflicting dependency."

2011-04-22T13:44:00.011+02:00

When plug-in installation fails in newer versions of Eclipse (such as "e4"), you may have to deal with cryptic error messages such as this one:

Cannot complete the install because of a conflicting dependency.
 Software being installed: EPIC 0.6.39 (org.epic.feature.main.feature.group 0.6.39)
 Software currently installed: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
 Only one of the following can be installed at once:
   Core Runtime 3.6.100.v20101122 (org.eclipse.core.runtime 3.6.100.v20101122)
   Core Runtime 3.7.0.v20110110 (org.eclipse.core.runtime 3.7.0.v20110110)
 Only one of the following can be installed at once:
   Equinox Launcher Linux X86 Fragment 1.1.100.v20110321 (org.eclipse.equinox.launcher.gtk.linux.x86 1.1.100.v20110321)
   Equinox Launcher Linux X86 Fragment 1.1.100.v20101018 (org.eclipse.equinox.launcher.gtk.linux.x86 1.1.100.v20101018)
 Only one of the following can be installed at once:
   Equinox Java Authentication and Authorization Service (JAAS) 1.1.0.v20110411 (org.eclipse.equinox.security 1.1.0.v20110411)
   Equinox Java Authentication and Authorization Service (JAAS) 1.1.0.v20101004 (org.eclipse.equinox.security 1.1.0.v20101004)
   Equinox Java Authentication and Authorization Service (JAAS) 1.1.0.v20110124-0830 (org.eclipse.equinox.security 1.1.0.v20110124-0830)
 Only one of the following can be installed at once:
   Refactoring UI 3.5.100.v20101012-0800 (org.eclipse.ltk.ui.refactoring 3.5.100.v20101012-0800)
   Refactoring UI 3.6.0.v20110307-2000a (org.eclipse.ltk.ui.refactoring 3.6.0.v20110307-2000a)
   Refactoring UI 3.5.100.v20110111-0800 (org.eclipse.ltk.ui.refactoring 3.5.100.v20110111-0800)
 Only one of the following can be installed at once:
   Eclipse Platform 4.1.0.v20110420-1420 (org.eclipse.platform 4.1.0.v20110420-1420)
   Eclipse Platform 4.1.0.v20110415-1401 (org.eclipse.platform 4.1.0.v20110415-1401)
   Eclipse Platform 3.7.0.v201104210100 (org.eclipse.platform 3.7.0.v201104210100)
   Eclipse Platform 3.7.0.v201103101119 (org.eclipse.platform 3.7.0.v201103101119)
   Eclipse Platform 4.1.0.v20110407-2200 (org.eclipse.platform 4.1.0.v20110407-2200)
   Eclipse Platform 4.1.0.v20110415-1030 (org.eclipse.platform 4.1.0.v20110415-1030)
   Eclipse Platform 3.7.0.v201103220800 (org.eclipse.platform 3.7.0.v201103220800)
   Eclipse Platform 3.7.0.v201104061223 (org.eclipse.platform 3.7.0.v201104061223)
   Eclipse Platform 4.1.0.v20110412-2200 (org.eclipse.platform 4.1.0.v20110412-2200)
   Eclipse Platform 3.7.0.v201104121532 (org.eclipse.platform 3.7.0.v201104121532)
   Eclipse Platform 3.7.0.v201104211800 (org.eclipse.platform 3.7.0.v201104211800)
   Eclipse Platform 4.1.0.v20110127-2200 (org.eclipse.platform 4.1.0.v20110127-2200)
   Eclipse Platform 3.7.0.v201102150800 (org.eclipse.platform 3.7.0.v201102150800)
   Eclipse Platform 4.1.0.v20101202-1530 (org.eclipse.platform 4.1.0.v20101202-1530)
   Eclipse Platform 4.1.0.v20110312-1300 (org.eclipse.platform 4.1.0.v20110312-1300)
   Eclipse Platform 4.1.0.v20110325-1411 (org.eclipse.platform 4.1.0.v20110325-1411)
   Eclipse Platform 4.1.0.v20110419-2200 (org.eclipse.platform 4.1.0.v20110419-2200)
   Eclipse Platform 3.7.0.v201012081300 (org.eclipse.platform 3.7.0.v201012081300)
   Eclipse Platform 4.1.0.v20110422-0200 (org.eclipse.platform 4.1.0.v20110422-0200)
   Eclipse Platform 4.1.0.v20110421-0500 (org.eclipse.platform 4.1.0.v20110421-0500)
   Eclipse Platform 4.1.0.v20110414-2200 (org.eclipse.platform 4.1.0.v20110414-2200)
   Eclipse Platform 3.7.0.v201104191004 (org.eclipse.platform 3.7.0.v201104191004)
 Cannot satisfy dependency:
   From: Equinox p2 Provisioning for IDEs 2.1.0.v20110228-897TFncFdHjO2qRuCh_UvYT (org.eclipse.equinox.p2.user.ui.feature.group 2.1.0.v20110228-897TFncFdHjO2qRuCh_UvYT)
   To: org.eclipse.equinox.security [1.1.0.v20110124-0830]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.r20110302-9gF7SHCIFt6ms-lIjrC6vK_XO-IabJMKu (org.eclipse.platform.feature.group 3.7.0.r20110302-9gF7SHCIFt6ms-lIjrC6vK_XO-IabJMKu)
   To: org.eclipse.platform [3.7.0.v201103101119]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20100923-9fF7MHDqFsAkplGz0n61z-yU57WHoz0JdMfaI (org.eclipse.platform.feature.group 3.7.0.v20100923-9fF7MHDqFsAkplGz0n61z-yU57WHoz0JdMfaI)
   To: org.eclipse.rcp.feature.group [3.7.0.v20101115-9FB-FqhFr3P05j0S-HRVgSR]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20110209-9fF7QHFQFsAlpyfO4rKw4lNW200gZcvGr (org.eclipse.platform.feature.group 3.7.0.v20110209-9fF7QHFQFsAlpyfO4rKw4lNW200gZcvGr)
   To: org.eclipse.ltk.ui.refactoring [3.5.100.v20110111-0800]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20110315-9gF7SHCJFt3cwDesZ5LrhrcdfooveV1uWqcqPF (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHCJFt3cwDesZ5LrhrcdfooveV1uWqcqPF)
   To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyIDVb_FRbwWP]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20110315-9gF7SHIZFt3cwDktE9WrW9XEj1iOiW9fgbhxME (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHIZFt3cwDktE9WrW9XEj1iOiW9fgbhxME)
   To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyImVoOcTaz-ZS]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20110315-9gF7SHLgFt4cwDmoWF-DvhVMEUrQfXBVSritR5 (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHLgFt4cwDmoWF-DvhVMEUrQfXBVSritR5)
   To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyIxIrTOSarWQ]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20110315-9gF7SHNFFt4cwDooaDrDsaVdEWi0oMLWlh6mbT (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHNFFt4cwDooaDrDsaVdEWi0oMLWlh6mbT)
   To: org.eclipse.equinox.security [1.1.0.v20110411]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20110315-9gF7SHNFFt4cwDqmhLPCtURdWWmWmO0Plh6mbT (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHNFFt4cwDqmhLPCtURdWWmWmO0Plh6mbT)
   To: org.eclipse.equinox.security [1.1.0.v20110411]
 Cannot satisfy dependency:
   From: Eclipse Platform 3.7.0.v20110315-9gF7SHNFFt4cwDrmlStE-Ufz0RVqQoc8RS8f2gQ (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHNFFt4cwDrmlStE-Ufz0RVqQoc8RS8f2gQ)
   To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyY_VrXOSeaUQ]
 Cannot satisfy dependency:
   From: Eclipse Platform 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU (org.eclipse.platform.feature.group 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU)
   To: org.eclipse.ltk.ui.refactoring [3.6.0.v20110307-2000a]
 Cannot satisfy dependency:
   From: Eclipse Platform 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU (org.eclipse.platform.feature.group 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU)
   To: org.eclipse.platform [4.1.0.v20110312-1300]
 Cannot satisfy dependency:
   From: Eclipse Product Configuration 1.0.0.I20110310-1119 (org.eclipse.rcp.configuration.feature.group 1.0.0.I20110310-1119)
   To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20101018]
 Cannot satisfy dependency:
   From: Eclipse RCP 3.7.0.v20101115-9FB-FqhFr3P05j0S-HRVgSR (org.eclipse.rcp.feature.group 3.7.0.v20101115-9FB-FqhFr3P05j0S-HRVgSR)
   To: org.eclipse.core.runtime [3.6.100.v20101122]
 Cannot satisfy dependency:
   From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyIDVb_FRbwWP (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyIDVb_FRbwWP)
   To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
 Cannot satisfy dependency:
   From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyImVoOcTaz-ZS (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyImVoOcTaz-ZS)
   To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
 Cannot satisfy dependency:
   From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyIxIrTOSarWQ (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyIxIrTOSarWQ)
   To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
 Cannot satisfy dependency:
   From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyY_VrXOSeaUQ (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyY_VrXOSeaUQ)
   To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
 Cannot satisfy dependency:
   From: Eclipse Project SDK 4.1.0.v20110303-7T7fA7F8Yw_bVbCrFz-gFceimF7IZRyogX1MAdUyRWqYu (org.eclipse.sdk.feature.group 4.1.0.v20110303-7T7fA7F8Yw_bVbCrFz-gFceimF7IZRyogX1MAdUyRWqYu)
   To: org.eclipse.platform.feature.group [4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU]
 Cannot satisfy dependency:
   From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
   To: org.eclipse.equinox.p2.user.ui.feature.group [2.1.0.v20110228-897TFncFdHjO2qRuCh_UvYT]
 Cannot satisfy dependency:
   From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
   To: org.eclipse.rcp.configuration.feature.group [1.0.0.I20110310-1119]
 Cannot satisfy dependency:
   From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
   To: org.eclipse.sdk.feature.group [4.1.0.v20110303-7T7fA7F8Yw_bVbCrFz-gFceimF7IZRyogX1MAdUyRWqYu]
 Cannot satisfy dependency:
   From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
   To: toolingorg.eclipse.sdk.ide.configuration [4.1.0.I20110312-1300]
 Cannot satisfy dependency:
   From: EPIC 0.6.39 (org.epic.feature.main.feature.group 0.6.39)
   To: org.eclipse.platform.feature.group [3.2.0,4.0.0)
 Cannot satisfy dependency:
   From: toolinggtk.linux.x86org.eclipse.core.runtime 4.1.0.I20110312-1300
   To: bundle org.eclipse.core.runtime 3.7.0.v20110110
 Cannot satisfy dependency:
   From: toolingorg.eclipse.sdk.ide.configuration 4.1.0.I20110312-1300
   To: toolinggtk.linux.x86org.eclipse.core.runtime [4.1.0.I20110312-1300]

The usability problems with this error message are evident: excessive length, lack of focus, little clue for the developer about possible solutions (and even less for a mere user who is just trying to install a plug-in). Paradoxically, the best way to even know which parts of the error message are relevant at all is to have already solved the problem at hand. Having done just this, here are some possibly helpful inferences:

Chances that you are going to solve a problem of this sort as a normal user are slight. The error likely lies in dependency declarations contained in feature.xml descriptors that are being processed by the installer and you cannot/are not supposed to manually edit those. It's safe to assume that it's the developers' job to sort out this mess. The remaining bullet points address how.
The "Only one of the following can be installed at once" parts of the message are misleading and should be ignored as noise.
The "Cannot satisfy dependency" parts might be relevant, however most of them are bogus, resulting from error propagation, and should be ignored as noise as well.
The root cause is likely contained in the feature.xml descriptor of whatever it is that you are trying to install. In the above example my installation attempt concerns the EPIC IDE, and only the following short snippet from the bottom of the error message actually matters:
```
 Cannot satisfy dependency:
   From: EPIC 0.6.39 (org.epic.feature.main.feature.group 0.6.39)
   To: org.eclipse.platform.feature.group [3.2.0,4.0.0)
```
The above dependency comes from a line in content.xml, a file which is found within content.jar downloaded by the installer from the update site. In my case the relevant line of content.xml looked like so:
```
<required namespace='org.eclipse.equinox.p2.iu'
         name='org.eclipse.platform.feature.group'
         range='[3.2.0,4.0.0)'/>
```
The file content.xml is a generated one, so to fix the problem (by removing the line) the developer must track back to the input files that are under his control. In my case it was feature.xml, which contained the following "offending" snippet:
```
  <requires>
     <import feature="org.eclipse.platform"
                version="3.2.0"
                match="compatible"/>
  </requires>
```

Eventually, changing match="compatible" to match="greaterOrEqual" in feature.xml and rebuilding content.jar fixed the cause of the original messy error message, as was to be expected ("compatible" means only compatible within the major version, i.e. "3.x.x"). After the fix content.xml contains range="3.2.0", which is fine for installation in 4.1.0.

Don't use Perl to compute (much)

2011-04-05T19:11:00.004+02:00

There are times when we want to understand and optimize performance of Perl programs, and there are other times when we just want a program to run faster, fast. Having a computation-intensive task at hand (hint: nested loops, numbers, arithmetics) speaks for a brute force approach - simply rewriting the algorithm in C is likely to yield great benefits with little intellectual strain.

Here's a little case study: a Perl version of a program which performs some 360,000,000 arithmetic operations takes 135 wallclock seconds to execute. After moving the computation to C the execution time drops to 1.7 seconds - a speedup of more than 70. Hint: to keep life simple given such a case, don't even bother with perlxs or worry about efficient I/O... just port your code to C, invoke the C program via system() and communicate through files or stdin/stdout.

(Of course, the same insight certainly applies to any interpreted language, not just Perl.)

Fix hover display of target links in Firefox 4

2011-03-22T18:36:00.007+01:00

Problem: After upgrading to Firefox 4 the first "improvement" that you'll notice is possibly the all-too-distracting popup which displays the full target URL whenever you move the mouse pointer over any link. Apparently they got rid of the status bar, but decided that for security's sake the information must still be displayed somewhere and chose a popup. Unfortunately, a popup which keeps appearing and disappearing over the main page display area feels much more intrusive than a fixed status bar area (which you could also hide altogether).

Solution: Install the poorly named add-on Status-4-Evar. It will instantly restore the original status bar without any further configuration.

And by the way, you can also disable "View > Toolbars > Tabs on Top" to restore the original proper location of tabs. To move the Home and Reload buttons back to where they belong left of the address entry field, you can right-click on the little arrow within that field (or almost anywhere else on the toolbar), select "Customize" and then drag the buttons back.

The most comfortable GUIs are the ones we're used to.

Fullscreen Flash Player on Debian lenny

2011-01-10T20:01:00.004+01:00

Problem: Flash plugin crashes in Firefox on 64 bit Debian lenny whenever entering fullscreen mode. The console displays an error message like "libflashplayer.so: undefined symbol: gtk_widget_get_window". Both Firefox and Flash plugin were installed manually rather than from a Debian package on the affected system.

Solution: This problem occurs due to an outdated version of 32 bit libgtk2 in Debian lenny (ia32-libs-gtk), which doesn't have the required symbol. As a remedy, you should install 64 bit versions of libgtk2, Firefox and Flash player from Debian packages. However, these packages are not found in lenny repository, so you first need to add the backports repository to your apt (important: also add contrib behind main in the repository definition!) After that, just go ahead with

apt-get -t lenny-backports install flashplugin-nonfree iceweasel libgtk2.0-0

Building scalable web applications

2011-01-06T19:32:00.005+01:00

I stumbled upon a presentation by Pressflow founder David Strauss with a nice checklist of common (open source) technologies and buzzwords for building highly scalable web applications. The slides focus on deployment and load/configuration management rather than software design considerations: Scalable LAMP infrastructure.

Surface drilling Drupal Panels 3

2011-01-05T01:47:00.002+01:00

This article contains some notes from a short reverse-engineering session targeted at Panels 6.x-3.8 and Chaos Tools 6.x-1.8. The goal was to capture some first impressions and record them for potential future exploration and understanding of that module. The motivation came from the current lack of technical documentation from the powers that be.

Before deploying the drills, I wish to share with you a few contemplative remarks about Drupal documentation. I feel disappointed by the supposedly "user-friendly" and sometimes overenthusiastic case-oriented explanations so prevalent in descriptions of Drupal ("if you want X, click on Y... how cool!"). Frankly, I find them much inferior to what I like to call model-oriented explanations ("the system consists of parts X, which have state Y and interact according to rules Z") based on strict terminology and aiming to identify intermediate causes at work. A good model capitalizes on your prior software engineering and site building experience by allowing you to generate tons of individual cases in a reasonable amount of time. On the other hand, a set of cases helps you solve only a limited number of problems quickly by imitation. A large database of cases takes much effort to collect, maintain and remember, and it does a poor job at eliminating residual risk. When you're hit with a problem not yet documented in the database and forced to somehow augment your existing cases, this task is going to take an extraordinary amount of time or require to call in heavy artillery (a consultant). In contrast, a well-constructed model takes just a little effort to acquire (but not to create!) and promises much more substantial risk reductions; model refinements do not require quantum leaps of imagination. By abstracting well, you can keep most issues out of your mind most of the time, and with a good choice of metaphors, related pieces of a solution should come together quite effortlessly. In effect, adopting a cohesive model-oriented approach allows you to deliver more consistent problem solving for your clients. Similarly, when you hire someone, they better bring this secret weapon in their arsenal rather than just Google + d.o.

Back to the main topic. A panel is a named, persistent set of specifications that govern a joint presentation of multiple pieces of content on the same page. These specs cover

which kinds of content or even which individual pieces of content should be retrieved from the database;
where (in which pane of a multi-pane layout) each piece of the retrieved content should appear;
how each piece of content should be rendered;
under what circumstances and for whom each piece of content should (dis)appear;
(for panel pages) which URI paths should trigger displaying the specified content.

A panel is defined by a site builder. It is stored either in the database (by default) or in code (when exported, e.g. for the sake of VCS). It is then used by Drupal as a blueprint for page construction whenever its associated path is requested by the browser. The specifications that comprise a panel are further subdivided into a (non-empty) set of variants, each of which may differ from the other variants in certain respects while sharing the common base configuration to foster reuse. (However, variants are unlike displays from Views in that they all have the same rank; there is no default variant that others would inherit from and override.)

Panels are somewhat at odds with Drupal core, meaning that they seek to replace rather than just expand upon and integrate with some of the previous ideas:

In their capability to lay out pieces of content, panels build upon—but also to an extent compete with—page templates defined by a theme. In particular, panes of a panel layout are conceptually equivalent to regions of a page template. Because a panel is placed within a region, it can be also thought of as a means for subdividing the region into multiple sub-regions (which are not known to the theme subsystem).
In their capability to retrieve and render pieces of content, Panels developers abandon Drupal's core idea that all content should be represented as either nodes (for user-defined content) or blocks (for reusable pieces of UI). Other kinds of HTML content, not belonging to either category and possibly computed on-the-fly, can also be made available to and rendered in a panel.

To put the above and following explanations in context, consider the following screenshot of the Panels UI (with some section numbering added):

The configuration sections are subdivided into three main areas: Summary, Settings, and Variants. The information under Summary and Settings concerns the panel as a whole, while the subsections under Variants pertain to each of the defined named variants. With a few exceptions, the section labels are not self-explanatory at first glance, and the information contained within each section cannot be easily understood separately from the other sections. In this sense, the UI is non-intuitive. It won't teach what can be done with panels in a learning-by-doing approach. It will cause concern with regard to unexplained settings and overly abstract terminology. The module author's line of defense is that you don't have to understand every piece of the UI to successfully use panels for simple layout scenarios. While true, the suggested adopt-now-explore-later approach also conveys an ominous message: if the documentation is not ever written—and if the adoption or future development of the module suffers—you may find yourself in a bad position by having placed bets on it. (It is not much consolation for new users that the friendly competition doesn't fare much better with regard to making their alternative comprehensible and that the Panels module is well-entrenched with a significant user base.) The question is: can we in the meanwhile remedy this situation by sacrificing some time on the altar of open source and simply peeking into Panels code? Or is the code as poorly done as the documentation? The rest of the article is my first shot at answers (for the impatient: yes and no, respectively). I start by exploring some of the more confusing—while at the same time important—sections of the Panels UI and conclude with a brief foray into code design and implementation issues.

It turns out that the key to understanding Panels configuration are the concepts of "contexts" and "content". These technical terms are not at all related to the Context module (best erase it out of your mind while studying Panels), only weakly related to the content types of the CCK module, and they might not match any colloquial meaning taken from a dictionary. Such is the nature of terms unleashed on the unsuspecting world at night by well-meaning software developers. Let's focus on "contexts" first. In Panels terminology, the term context refers to a specification of a piece of (renderable) content that should be loaded and become known under a chosen id within the panel variant before it is output as HTML. For example, if your panel should display three nodes next to each other, it would obviously first have to load them, which you would specify by adding three contexts of type node to some panel variant. The Contexts section (2) lets you directly specify the pieces of content that should be loaded; for example specify a node by its nid or title. Alternatively, you could parametrize the panel to load different pieces of content depending on what has been provided through the URL. To achieve this, you would include placeholders in the panel's path (as shown on the screenshot) and then use the Arguments section (1) to associate each placeholder with a context type (e.g. specify that the first placeholder called %node1 really refers to a node ID rather than a taxonomy term ID; at request handling time, the URL is first taken apart by Drupal's menu subsystem and for each placeholder all modules, including Panels, are consulted – see _menu_load_objects in includes/menu.inc for details). Finally, by using the Relationships area of the Contexts section (2) you could also specify a context that loads a piece of content related to another, previously specified, piece of content (say, load the user who is an author of the second node loaded by the node context). Based on the preceding description, a better term for "contexts" might have been "content loaders"; alas, the release night is over, and it is too late for changes now.

Now let's turn to content. In Panels terminology it refers to "something that can be placed (and somehow rendered in) a pane" through the Content section (3). Unsurprisingly, the pieces of content loaded by contexts as well as their individual parts (such as CCK fields of nodes; or the user picture from a profile) fall into this category. For this reason you are generally well-advised to first define contexts before even turning to the Content section. However, other pieces of content (such as blocks or HTML snippets) are also available without the need (or possibility) to obtain them through a context. The advantage of loading panel content through contexts is that the attributes of the loaded objects then become available within all panes as variables that can be used in string substitutions (e.g. %node1:title or %user1:name, where the prefix equals context id). For example, having established the three node contexts and one user context as mentioned above, you could add a HTML snippet (aka "custom content") with the node titles and the user name to a pane.

Having acquired a basic understanding of the context/content pair of concepts and experimented a bit with the Panels UI, here are the next few natural followup questions that come to mind:

Where do all the selectable context and content types in the Panels UI (and their configuration forms) come from?
How does the Panels module know to offer "Node", "Taxonomy term" and "User" contexts and how do you know what data exactly they load?
What the heck is a "Node add form" context supposed to load and how is it different from "Node"?
What determines which items appear on the "Miscellaneous" tab when adding content as opposed to the "Widgets" tab?
What if you want to add something that is not at all listed in Context or Content sections?
How exactly is the Relationships combo populated?
Should we degrade ourselves to click monkeys and figure out all of the above by trial and error?

I won't even come close to providing detailed (boring?) answers to all of the above questions and will instead address the last (somewhat rhetorical) one by providing a little push in the right direction. As announced before, the solution is sought (and found) by examining the code. It turns out that most of the items and forms you come across while interacting with the Panels UI are not hard-coded anywhere in the Panels module, but rather contributed dynamically by code. "I know – panel hooks!" might be your first guess, but unfortunately it's not that simple. There is a lot more indirection in the Panels code than meets the eye and a more refined mechanism than good old Drupal hooks is at play. Furthermore, it turns out that neither contexts nor content types are designed to be panel-specific. In fact, most of them are not even implemented in the Panels module, but rather belong to the Chaos Tools module, upon which the Panels module depends. In general it's safe to say that the indirection and abstraction sauce is good for flexibility and extensibility and also a definitive poison for code comprehension. The case of Panels is no exception, and so I guess that the author might be playing with fire to some extent ("we have this wonderfully flexible super-duper API for every conceivable task, which unfortunately noone besides us can wrap their head around). I'm glad to report that I have seen much worse and that the code design and inline documentation are definitely more solid than what you see in "advanced help" today. In short, you can wrap your head around it.

To navigate around Panels code—and to see how the administrative UI is built and what each selectable item stands for—you have to first grasp the much more general concept of CTools Plugins (reified hooks). This part of the Chaos Tools module is an example of the Strategy pattern at work. It aims to facilitate a division of labor between module developers and site builders. A plugin instance is an exchangeable persistent object responsible for performing a set of tasks in its own particular way. Plugins are typically selectable (and so instantiated) through the UI of those Drupal modules that depend upon them (like Panels or Views). They may also contain callbacks that construct an administrative form-based UI to allow site builders detailed configuration of each plugin instance. Alternatively, plugins may be organized hierarchically (in a type:subtype or parent:child fashion), for the benefit of a more structured selection UI. The exact set of responsibilities for a plugin type is usually specified by the module developer who invokes plugins of that type from her module. Particular implementations of those responsibilities are then provided as plugins by (potentially) different developers. A plugin is usually implemented in a single .inc file. Plugin names must be unique within a plugin type, but do not need to be unique across types. Plugins are used by modules or by each other. The CTools library acts as an intermediary and a directory service for publishing, locating and invoking plugins. This terminology and the discussed relationships are illustrated by the following examples:

Plugin type	Responsibility of plugin type	Plugin	Plugin functionality	Plugin implemented in	Plugin used by	Plugin instance (examples)
contexts	Provide renderable content for a panel.	node	Load and provide data from a node specified by nid or title.	ctools/plugins/contexts/node.inc	panels.module (indirectly, through its own plugin panels_context of type task_handlers)	node1 and node2 in Variant1 of panelpage1
"	"	user	Load and provide data from a user profile by uid or name.	ctools/plugins/contexts/user.inc	panels.module (indirectly, through its own plugin panels_context of type task_handlers)	user1 in Variant1 of panelpage1
content_types	Render available content within an individual panel pane.	node	Render available node content.	ctools/plugins/content_types/node.inc	panels.module (indirectly, through its own plugin standard of type display_renderers)	"Node 1: ID" content in left pane and "Node 2: ID" content in right pane of Variant1 of panelpage1
"	"	user	Render a picture from user profile.	ctools/plugins/content_types/user_context/user_picture.inc	panels.module (indirectly, through its own plugin standard of type display_renderers)	"Node1 author" user picture in Variant1 of panelpage1

The required naming conventions and steps for defining plugins (and plugin types) and invoking plugin instances are documented in "advanced help" shipped with the CTools module, which you are advised to read carefully. The CTools library is used pervasively and consistently throughout Earl Miles's (aka merlinofchaos) modules (and also in some others). Thus, if we view Drupal as a domain specific language, CTools should count as a potentially prominent dialect. Becoming familiar with it will not only help you understand Panels, but also Views and potentially future of Drupal core, if it should spread into more mainstream use.

Note: while looking at the CTools module, I also found that it supports third-party module developers in evolving and versioning their own modules' APIs, including catering to different client modules using different versions of an API at the same time. The term API here refers to the set of hooks which the developed module invokes (i.e. the hooks provided [not the hooks implemented or library functions offered!] by that module). When CTools versioning support is utilized by a module, hook implementors must partition their code into multiple files, each of which corresponds to a particular version of the API. When the hook provider is about to invoke a (particular version of) a hook, it can query CTools for all matching implementations, while excluding any non-matching (e.g. too old or too new) implementations. This mechanism, documented in "e;advanced help", appears to be almost unused, and probably will remain so due to its confusing official description. Nevertheless, it is interesting to see an idea of this sort in place, given Drupal's track record of neglect for backwards-capability (at least between major versions).

Drupal multi-sites in a nutshell

2010-12-30T16:27:00.003+01:00

Although Drupal multi-sites are conceptually trivial, I found no satisfactory (short!) explanation in the official documentation or on the web. You can listen to this half an hour long presentation like I did, but I'd rather have read the following summary. I suppose you're already familiar with single-site Drupal setups and would like to understand multi-site now:

A multi-site configuration consists of a single code base along with multiple databases. Essentially, there is one database per site, although relocating some shared tables to a common database (in the same database server) is also possible. This approach should be contrasted with a shared-nothing "multiple site" configuration.
The key to the entire concept is how Drupal locates the settings.php file based on the requested URL. You can steer it to a different settings.php (hence, a different database) for any combination of hostname, port and URI prefix. The exact lookup algorithm and naming conventions are described in settings.php itself.
There is no "master" configuration or "master" site, although the "default" site is used as fallback when a HTTP request matches no specific site (the normal case for non-multi-site setups). Except for that, all sites have the same rank and are unaware of each other. Each site uses the same core code and also the same sites/all/modules and sites/all/themes, while it may also add its own sites/sitedirectory/modules and sites/sitedirectory/themes. If the same module or theme is in present both locations, the site-specific location takes precedence, so you can have diverging versions of modules or themes between sites.
To add another site (or convert your single-site setup to a multi-site setup), simply create a new database for it, add sites/sitedirectory, copy sites/sitedirectory/default.settings.php to sites/sitedirectory/settings.php and visit the new site's URL in the browser. The usual Drupal installation procedure will be offered.
The files folder is usually placed under the site-specific folder, i.e. separate for each site (if you want it shared, you should also take care to share the files database table and possibly others).
Because the databases are separate, each site can have a different subset of all the available modules/themes enabled.
Because the databases are separate, you have to run update.php separately for each site when upgrading Drupal core or contributed modules.
Because the Drupal core is shared, you may run into dependency problems if at least one site requires a different version of Drupal core than the others.
Because the web server user and file permissions are shared, the multi-site configuration is potentially insecure: any site administrator who can run PHP code may access files (and, by peeking into settings.php, also gain full access to the database) of any site.

Still uneasy about SRP

2010-12-19T15:13:00.002+01:00

I originally intended this as a short comment to SRP, as easy as 123, but since that blog won't let me post it as comment for some reason, I decided to put it here. Go read the original article for context. Essentially, the author wishes to enlighten us about the proper interpretation of the Single Responsibility Principle, which is one of the many "tried and true" pieces of wisdom promoted by OO gurus.

While the considerations described in the article are all fine and a nice example of the kinds of reasoning involved in building software, it also exposes a real problem with the "principle" as such, and it's a pity that it doesn't hit the target more directly.

It turns out plainly that the principle cannot be interpreted or applied without a good deal of informed speculation about what is going to happen in the future. Whether or not a particular structuring of code is good (enough) depends heavily on that future stream of events and interactions, some of which are more likely to occur than others (and their probabilities better be conditioned on individual case circumstances). However, notice how the principle misleads people to think solely in terms of some mystical absolute characteristics of the programming language classes. The result are pointless and confusing discussions on that shallow "ontological" level (is my class more responsible than yours? and how exactly do you count "reasons for change"?). The same kind of insidious error also hides in other OO principles ("is class X more abstract than class Y?").

In conclusion, it would be much easier to discuss matters not in terms of perceived (or misperceived) structural properties, but simply by addressing the known future risks and possible strategies of risk avoidance, some of which may be implemented in structure, others in the engineering process or tools. For example, there is also a risk associated with dividing code into unnecessarily many interacting modules (or deep class hierarchies), especially if the future maintainers lack adequate tools to piece together what the developer has pulled apart. Furthermore, there is also an immediate cost to making the structure more amenable to change or extension; if the expected kinds of change never happen in the future, you have wasted effort. The problem with SRP is that it tends to focus your mind on one thing and forget these alternate risks (at least for a while), and the problem with principles in general is that they may foster irresponsibility or magical thinking ("if only we obey the Rules, everything will go well; and if something hasn't gone well, we must have disobeyed the Rules").

Software design comes down to decision making under uncertainty, and software engineering principles are just an attempt to capture some aspects of that process using short catchphrases. They may serve us well as reminders if we're already capable of the kind of complex decision making required, but they are not a substitute for careful thinking and a poor learning aid. In the end, the narrator of the original article has demonstrated his usefulness to his colleague, while SRP has not.

Restricting CCK forms in Drupal 6

2010-12-11T18:35:00.004+01:00

This article contains a programming howto for stripping unwanted fields from forms generated by the CCK module in Drupal 6. It is natural for a content type to gather as many fields as the logical data schema requires (essentially, the decision about adding a field to an existing content type or creating a new content type, is dictated by considerations about data cardinality and independence in life cycles of individual pieces of data). Real-world applications will usually require that certain subsets of these fields should not be editable by everyone, or that they should only be editable at certain times or when certain conditions are met (for example, based on another field's value within the node being edited). Understandably, CCK doesn't know anything about such special needs out-of-the-box, and so it will render all node fields as equals on a common form.

Consider a case where you have already created a complex default form for editing the entire content type, possibly with field groups and a custom theme which renders them in a fancy hierarchical way. Now let's say you have another user role which is only supposed to understand and edit a few fields from the said content type. You want a new form which displays only those fields and saves the submitted values to the node, validating them in exactly the same way as they would be in the full-node edit scenario. To make things even more interesting, suppose that one of the fields is a select box for which you want to restrict the list of allowed values when it is presented on the restricted form.

The correct solution is short in code, but rather tricky conceptually. It requires at least an intermediate level of understanding of the Drupal Form API and theming layer. The following steps can be used as a guideline:

Choose a path for the new form. It is a logical choice to extend the default node edit path. That is, let /node/1234/edit refer to the default form and let /node/1234/edit/custom refer to your custom form. In your module, implement hook_menu() and copy the item from node.module which refers to /node/%node/edit. At this point you have an opportunity to turn it into a MENU_CALLBACK (if you don't want the new form to be accessible through the standard tabs) and also to restrict access by defining a new access callback. Don't forget to specify 'file path' => drupal_get_path('module', 'node'), otherwise the file node.pages.inc defined by node.module would not be found. The new item should look like so:
```
  $items['node/%node/edit/custom'] = array(
    'title' => 'Edit custom',
    'page callback' => 'node_page_edit',
    'page arguments' => array(1),
    'access callback' => 'yourmodule_edit_custom_access',
    'weight' => 1,
    'file' => 'node.pages.inc',
    'file path' => drupal_get_path('module', 'node'),
    'type' => MENU_CALLBACK,
  );
```
Implement hook_form_alter() to disable access to unwanted fields on your custom form. You will also probably want to set a distinct theme function for the new form, so that you can arrange the wanted fields in a sensible way. The code in yourmodule_form_alter should look like so:
```
  if ($form_id == 'yourcontenttype_node_form' && arg(3) == 'custom') {
    $form['#theme'] = 'yourmodule_custom_form';
    $allowed_fields = array('field_first', 'field_second', 'field_third');
    foreach ($form as $key => &$field) {
      if (strpos($key, 'field_') === 0 && !in_array($key, $allowed_fields))
        $field['#access'] = FALSE;
    }
  }
```
The original form might also contain other form elements that need to be disabled and whose names don't begin with field_. You can find out what they are visually by looking at the form and by dumping the keys of $form. One good example is the buttons key, which contains the submit, preview and delete buttons. It is possible that some form elements contributed by other modules are not yet present when your hook_form_alter() is called. In this case, you will likely have to implement another hook_form_alter() in another module and set this module's weight to a big value in the system table, to ensure that it executes last.
Register the theme function yourcustom_form specified above via hook_theme() and also provide an actual implementation in a function called theme_yourmodule_custom_form, which receives $form as argument. Alternatively, you could declare it in hook_theme() as a template like so:
```
  'yourmodule_custom_form' => array(
    'arguments' => array('form' => NULL, 'user' => NULL),
    'template' => 'custom_form',
  ),
```
and then create a template file custom_form.tpl.php in your theme folder.
Within your theme function or template, render the whole form or individual fields like so:
```
  // Whole form
  // print drupal_render($form);

  // Individual fields
  print drupal_render($form['fieldgroup_tabs']['group_somegroup']);
  print drupal_render($form['buttons']['submit']);
```
In general, if you cannot render the whole form at once because it would result in unwanted artifacts (e.g. unwanted group headers from the original form or some such), it is a good approach to render the wanted fields individually first and then call drupal_render($form) within a display:none div element, which will only output the as-of-yet unrendered fields. This will also ensure that any hidden fields are output at all. Mind the possible security implications; if you failed to set #access to FALSE on some fields in the step above, you'd get invisible, yet still updateable fields (the user might use Firebug to get at them). Accordingly, only rely on the display:none trick for visuals, but not as a method of concealing fields.

In the problem statement a requirement to restrict allowed values for a particular field was mentioned. This can be done through the #afterbuild callback installed on the field in question, like so:

  // In hook_form_alter() if branch: 
  $form['field_first']['#after_build'][] =
     'yourmodule_field_first_after_build';
  // ...

  function yourmodule_field_first_after_build($element, &$form_state) {
    unset($element['value']['#options']['']);
    return $element;
  }

A word of caution: if you are using a checkbox CCK field in your content type, you might to run into a bug when you set #access to FALSE, which would prevent the custom form from validating. As a workaround, two patches for optionwidgets.module were available at writing time (pick your favorite).

In general, when working with form data structures, remember that the content of these structures varies considerably depending on the current processing stage within the form engine. (Unfortunately, these variations and the separation between API and internals are rather badly documented, which makes it easy to write unstable code.) It's also good to realize that FAPI's "form elements" are hierarchical in nature and act as a ~~mind-numbingly complicated~~ powerful adapter layer between the database fields and HTML form inputs. In particular do realize that a single FAPI form element might expand into multiple form inputs and/or supply values for mutliple database fields.

For further information, I refer you to the following sources:

Creating custom elements using Drupal 6 (besides of explaining custom element types, it also goes into the FAPI processing logic)
FAPI workflow illustration
FAPI reference

Solution for unavailable CCK fields in Drupal 6 Views

2010-11-13T22:03:00.004+01:00

Problem: Using Drupal 6 Views UI, you want to add a particular CCK field (say, as a filter), but this field seems to be not offered at all for some reason. Because most Drupal code is not protected by assertions (amateur coding?), corrupt data is likely to flow far before a visible problem is eventually triggered.

Solution: My particular trouble was caused by a Drupal 5 to Drupal 6 CCK migration bug concerning number_integer fields. However, given the general nature of this problem I decided to record some general troubleshooting tips here.

One thing to check for is metadata corruption - the Views module might be working properly, but the CCK module might not be delivering a correct description of the affected field for some reason.

As a first guess, examine the field's entry in the content_node_field table (and perhaps also content_node_field_instance), particularly look at the db_columns attribute. Does it appear unusual when compared to other, correctly working fields?

If you are out of luck so far, here are two locations in code where to install debug output to gather more information:

Right after the call to views_fetch_fields in function views_ui_add_item_form - does your field appear in the $options array, as it should? (probably not)
content_views_field_views_data - how does the returned structure for your field compare to those of other (working) fields?

To trigger calls to these function interact with the Views UI (e.g. click the add filter button), but do not forget to /devel/cache/clear each time! Otherwise the critical code paths will not be traversed due to caching.

Why Drupal seems slow

2010-11-06T19:52:00.004+01:00

The minimum response time metric (rather than, say, average response time) does a good job of describing whether a web site feels slow or snappy. Below are two pictures that illustrate how the mighty Drupal 6 is inferior in this respect to the much simpler Website Baker CMS. These measurements were taken for a typical web page filled with some images and a moderate amout text in case of Website Baker, and on an almost empty page for Drupal 6 with an installed Ubercart (shopping cart) suite of modules (with no caching). KCachegrind was used for visualization:

Website Baker 2.7 response time (under profiler): 60 ms Drupal 6 response time (under profiler): 510 ms

Also compare the number of "boxes", which reflect nested procedure calls in both cases.

Does it mean that given a choice you should use Website Baker instead of Drupal? The answer obviously will depend on the required features. The main insight here is that this overhead is very unlikely to go away in the future. It arises from Drupal's overall architecture involving interactions among many small modules, most of which are contributed by third parties and neither explicitly tested nor certified for performance. While offering lots of functionality and extensibility out-of-the-box, Drupal puts much strain in terms of the required quality assurance on the application designer - with performance being just one aspect of the challenge.

Update 2011/01: the latency of rendering simple pages apparently got even worse in Drupal 7 (see comments to this article). I'm aware that it is rather easy to boost Drupal performance for anonymous, non-targeted content several orders of magnitude just by using a reverse proxy cache and so reduce response time to equal network latency. However, I doubt that an average small Drupal site would be able to configure and deploy such optimizations.

Slow rendering of large tables in Firefox

2010-11-06T19:23:00.004+01:00

A few performance measurements of rendering a table with 15 columns and 1927 rows (a dense matrix with styled cell content) in different browsers:

MSIE 7.0	13,6s
Opera 10.62	13,9s
Firefox 3.6.8	40.3s

Firefox, you're doing it wrong!

Solution for XHR ERROR: Response_Code = -1 Error_Message = Request Error

2010-08-03T14:40:00.003+02:00

Problem: When trying to open a URL using Selenium RC 1.0.3 with Firefox 3.6, the following error is reported through the server log:

XHR ERROR: Response_Code = -1 Error_Message = Request Error

after which the browser windows disappear and the session is aborted. Interestingly, this only happens when trying to access remote sites, not sites from localhost.

Workaround: This error seems to be caused by something in the Firefox profile template provided with option -firefoxProfileTemplate. I temporarily edited selenium-browserbot.js within the selenium-server.jar to make it ignore the error (new line highlighted in red):

    this.isNewPageLoaded = function() {
        if (this.pageLoadError) {
            LOG.error("isNewPageLoaded found an old pageLoadError");
            var e = this.pageLoadError;
            this.pageLoadError = null;
            throw e;
        }

        if (self.ignoreResponseCode) {
            return self.newPageLoaded;
        } else {
            if (self.isXhrSent && self.isXhrDone) {
                if (self.xhrResponseCode == -1) return self.newPageLoaded;
                if (!((self.xhrResponseCode >= 200 &&
                    self.xhrResponseCode <= 399) ||
                    self.xhrResponseCode == 0)) {

With this change, the error ceased to be reported. I could no longer reproduce the problem later after removing this temporary fix.

Perl memory profiling

2010-06-24T00:18:00.005+02:00

Perl's garbage collector uses reference counting to keep track of which data structures are "live" on the heap and which can be deallocated. This means that memory will never be freed if your data structures contain cyclic references, even when all variables referencing those data structures go out of scope. It is generally up to you as a developer to break (or "weaken") such references if you want timely memory deallocation. Otherwise, memory deallocation will happen when your script terminates, but that might be too late, especially in case of long-running processes. (This is a good reason to avoid long-running processes, by the way. Paritioning your code into scripts/processes is a practical example of a fault-tolerance technique known as "micro-reboots".)

To check the overall (maximum) memory footprint of a Perl interpreter, use valgrind --tool=massif perl yourscript.pl, then ms_print massif.out.pid. However, note that it slows down the execution significantly and adds even more memory overhead, so handle with care.

Here is a list of developer-oriented Perl modules that will help with troubleshooting memory leaks in Perl:

Devel::Gladiator shows you counts of all live objects on the heap, by object class/category, including leaked ones. This is somewhat similar to - but more limited than - the jmap tool for Java VMs. You can also get the actual references, which you can then pass to...
Devel::Size, which lets you calculate the amount of memory taken up by a data structure (one that is not yet leaked).
Devel::Cycle examines a data structure for cyclical references (if such exist, the data structure is bound to cause memory leaks).
Devel::FindRef helps figure out where references to a data structure are being held.
Devel::Events::Objects offers instrumentation for constructor/destructor invocations.
Devel::Leak gives you the total number of currently live data structures.

Finally, you may wish to pass the -Dm command-line switch to Perl. However, unlike the modules, it requires a version compiled with debugging support and gives low-level information that is difficult to correlate with application-level code.

Solution for very slow Win XP boot, black screen

2010-06-15T14:47:00.002+02:00

Problem: a HP desktop PC suddenly and consistently takes a very long time to boot. The HP invent logo stays for about 30 seconds when the machine is turned on (it should only flash briefly). Furthermore, the during Windows XP boot sequence the screen is blanked and stays black for more than a minute before the login prompt is displayed. Supplementary info: According to Bootvis, the boot process takes 130+ seconds. Moreover, in Bootvis CPU activity trace, CPU load is reported 100% during the black screen phase (with no disk activity). The same delays also occur when starting Windows in safe mode.

Solution: in this particular case, it was an external USB hard disk that somehow got stuck. Turning off the disk and turning it back again fixed both delays (HP invent logo and black screen), bringing boot to normal speed. Lesson learned: always first detach USB devices when troubleshooting this kind of a problem.

Solution for GnuTLS internal error in cadaver

2010-06-07T16:10:00.002+02:00

Problem: the Linux WebDav client cadaver (Debian 0.23.3-1 binary package or compiled from source) fails the handshake abruptly instead of asking for the client certificate configured with set client-cert. When compiled from source, both GnuTLS and OpenSSL versions fail, although with slightly different error meesages. It used to work. The following error message is displayed by the GnuTLS version:

Could not open collection:
SSL handshake failed, client certificate was requested: SSL error: GnuTLS internal error.

Solution: the issue seems to be with libneon 0.27 and/or GnuTLS library. Downgrade to cadaver 0.22.5, which uses libneon 0.26, to make it work again.

DTrace-enabled Perl in OpenSolaris

2010-05-31T21:07:00.009+02:00

This post contains a few state-of-the-art notes concerning Perl-specific DTrace probes available in OpenSolaris today (2010-05-31). For instructions how to quickly install a working DTrace-enabled version of Perl, skip to the next section.

The OpenSolaris (2010.03, snv_134) Perl binary package comes with DTrace support disabled. Probably a good thing because the current implementation incurs a reported 5% worst-case performance hit - even with no enabled probes!
Perl source packages from 5.12.x on (or maybe already in 5.10.x?) have some basic DTrace support, good for instrumenting calls to and returns from subroutines.
More probes were made available through a patch from Sven Dowideit. However, his patch does not apply cleanly to 5.12.x.
Building Perl from source with DTrace support fails due to a known bug in Makefile.SH.
Why bother? To see system calls and kernel (e.g. I/O) activity incurred by a particular Perl subroutine call, for example. Or to monitor memory allocations. For pure Perl performance profiling better tools exist (such as NYTProf).

Building DTrace-enabled Perl in OpenSolaris

Download a recent Perl source code package. I grabbed 5.12.1.
If you just want the subroutine entry/return probe, apply the mini-patch from Peter Bray's bug report to avoid a build error about missing op.o and perl.o, then skip the next step. On the other hand, if you'd like to try Sven Dowideit's probes as well:
Download and apply this patch (version 2) to your freshly unpacked Perl source. This is a version of Sven's patch updated by myself to apply cleanly to 5.12.x, with an additional fix to Makefile.SH to avoid build interruptions and a fix to avoid a strange segfault in Perl_ck_require when tracing of loaded modules is enabled. Please note that I have not tested Sven's patch extensively. We are probably talking about a development build here that may aid your Perl code debugging efforts rather than something to put in production.

To compile:

make distclean
./Configure -des -Dprefix="/opt/perl-5.12.1" -Dusedtrace -Duseshrplib
make

Testing whether it works

Generally, the new probes become visible when a Perl process is running. You should be able to see them with the following command:

pfexec dtrace -l | grep perl

Here is a little test Perl script:

#!/opt/perl/bin/perl

foo() for 1..50;

sub foo
{
  print BAD_HANDLE "In foo\n";
  sleep(1);
}

A little test DTrace script:

#!/usr/sbin/dtrace -s

perl$target:::sub-entry, perl$target:::sub-return
{
  printf("%s %s (%s:%d)\n", probename == "sub-entry"
      ? "->" : "<-", copyinstr(arg0), copyinstr(arg1), arg2); }

Running both of them together:

pfexec dtrace -Z -s ./script.d -c ./script.pl

Several nice sample scripts for DTrace-enabled Perl are included in the DTrace Toolkit. Check it out.

If you want to see Sven's probes in action: download, read and run his example Perl DTrace script. Alternatively, you might try my DTrace script to measure time consumed by use/require statements.

Web-based front-end to MySQL's EXPLAIN

2010-05-17T23:08:00.004+02:00

If you are troubleshooting slow MySQL queries and are confused with the tabular output of the EXPLAIN command, you may wish to check out my little MySQL EXPLAINer service. You upload a CSV file with whatever output EXPLAIN SELECT has given you, and the query execution plan are is presented in form of a tree.

The CGI script is actually just a web front-end to the nice command-line tool mk-visual-explain, which does all the hard work of parsing EXPLAIN output. All due credit goes to the developers of Maatkit - read their documentation and slides referenced from their site if you are really interested in the technical details.

Solution for MySQL query slow, index does not help

2010-05-11T17:00:00.006+02:00

Problem: a MySQL query containing many joined tables is extremely slow (more than 1 minute). Proper indexes are present and being used according to EXPLAIN. The number of processed and returned rows is small (the problem persists with a COUNT(*) query).

Diagnosis: you may be hitting a performance problem with the MySQL query optimizer, which computes an "optimal" execution plan before the query is executed. It is difficult to find out without instrumentation (see example below on how to measure the time spent in optimizer), but it is easy to try whether the recommended solution helps.

Solution: change the MySQL parameter optimizer_search_depth, e.g. SET optimizer_search_depth=0 in the session before executing the query.

How to measure time spent in the MySQL query optimizer

On platforms which support DTrace and with an appopriate version of MySQL (I tested with 5.1.46, OpenSolaris), you can use the following DTrace script:

#!/usr/sbin/dtrace -s
/*
 * Measures and shows time spent in the mysql query optimizer.
 *
 * Usage: ./mysql-optimizer-check.d -p `pgrep -x mysqld`
*/
#pragma D option quiet

pid$target:mysqld:*mysql_parse*:entry
{
    printf("Query: %s\n", copyinstr(arg1));
}

pid$target:mysqld:*JOIN*optimiz*:entry
{
    self->start = timestamp;
    printf("Query optimizer entered\n");
}

pid$target:mysqld:*JOIN*optimiz*:return
{
    this->elapsed = timestamp;
    this->nano = this->elapsed - self->start;
    this->time1 = (this->elapsed - self->start)/1000000000;
    this->time2 = ((this->nano % 1000000000) * 100) / 1000000000;
    printf("Time spent in optimizer: %d.%ds\n", this->time1, this->time2);
    this->elapsed = 0;
    self->start = 0;
}

To run this script in OpenSolaris:

pfexec ./mysql-optimizer-time.d -p `pgrep -x mysqld`

Sample output (first query executes instantly, second query hangs and is aborted by myself after 7.72s):

Query: select @@version_comment limit 1
Query optimizer entered
Time spent in optimizer: 0.0s
Query: SELECT
    DISTINCT appl_app_id_appl.appl as 'appl.app_id_foreign',
    appl.app_id as 'appl.app_id',
    appl_app_id_appl.app_id as 'appl_app_id_appl.app_id',
    appl_app_id_appl.mandant_id as 'appl_app_id_appl.mandant_id',
    appl_app_id_appl.appl as
Query optimizer entered
Query: KILL /*!50000 QUERY */ 9
Time spent in optimizer: 7.72s
^C

Troubleshooting GRUB hangs

2010-05-04T14:39:00.004+02:00

This post describes one possible scenario in which GRUB hangs (freezes) with the four letters GRUB and a blinking cursor displayed on the screen, even before the boot menu appears. It might also display "Loading stage2..." and then hang. You might or might not be able to restart the computer with CTRL+ALT+DEL when this occurs. An explanation and a procedure for identifying or excluding the scenario is provided.

When GRUB hangs before displaying the boot menu, the immediate cause most likely is a failed attempt to load the stage2 file from the root partition ("root" refers to whatever has been specified when installing GRUB into boot sector). If you get no error messages at all, just a blinking cursor as mentioned above, then the underlying problem may lie in the implementation of the BIOS interrupt 13h, on which the installed GRUB (i.e. stage1 stored in the boot sector) depends to read raw data from disk. GRUB cannot avoid the crash if its humble attempt to read sector n from disk leads to a locked-up computer. When this happens, some garbage characters might also appear on the screen (printed by BIOS, not GRUB).

It seems that some BIOSes, even versions released as late as 2005, have big trouble reading sectors (blocks) beyond a certain boundary. In my test case (AMIBIOS, 80 GB Maxtor disk) the last readable sector turned out to be 66059279.

In order to find out whether or not you are experiencing the same problem, you should perform these additional tests:

Check the sector number where your root partition begins and ends (using fdisk, type u to change display units to sectors). (If it is the first disk partition, it's unlikely that you have the problem discussed here.)
Insert a boot CD with a working GRUB menu (a simple /usr/sbin/grub shell is not enough; see note below). Press the key 'c' after the GRUB menu appears while booting from the CD. Now enter the command root (hdm,n), replacing m with the disk number (in Linux: 0 = hda, 1 = hdc) and n with the partition number (in Linux: 0 = hdm1, 1 = hdm2, etc.) on which the file /boot/grub/stage2 is supposed to be found. If it hangs immediately after entering the command, but does not hang for root (hd0,0), it is likely that you have the described problem.
You can refine the diagnosis further by attempting to read individual sectors using the command cat (hd0)sector_num+1. For example: cat (hd0)66059280+1 produced the error message "Error 18: Selected cylinder exceeds maximum supported by BIOS" in my case, attempts with a smaller sector number worked, and attempts to read a much higher sector number (where the actual root partition started) caused hanging. When you see this behavior, you can be quite certain, that you have a broken BIOS.

Note that performing the above tests from a grub shell after having successfully booted (say, to another OS on the same PC or through a Live CD) will not give the symptoms and therefore will not help troubleshooting. The grub shell binary uses different system calls to read sectors than the actual stage2 binary and these may work well where the native BIOS interrupt fails.

In my case, it didn't help to upgrade BIOS. However, repartitioning and placing the root partition into lower sectors (swapping hdc2 with hdc3) solved the problem... or so I thought.

In a cruel twist of fate, the problem re-occurred with exactly the same symptoms just a few days later. It turned out that I moved the root partition toward the disk's beginning, but did not resize it, so that the end still stretched beyond the fatal sector. Editing menu.lst or possibly simply rebooting (OpenSolaris) moved one of the files required by GRUB to higher sectors, making the partition unbootable again. Lesson: if you have the problem described here (which due to its nature you might only find out when your system suddenly and inexplicably stops booting), the whole partition which contains GRUB files must be contained in low sectors!

Update: the recommendation of moving the boot partition paritition into low sectors is apparently not enough with OpenSolaris. My machine won't boot again. The file system code in GRUB actually reads sectors that lie outside of the boot partition boundary. And even after hacking it stop reading such sectors, the final configuration doesn't boot. Although both kernel$ and module$ can be set and files are reported found, all I get on a boot attempt is a blinking cursor. I suppose some of the OpenSolaris kernel/boot loader may be trying to read the high sectors as well. This is where I say farewell to OpenSolaris (on that machine ;-)).

Windows GUI automation with AutoIt and Perl

2010-04-17T17:27:00.002+02:00

This post contains some notes and thoughts from a short evaluation session for AutoIt3, a scripting tool that can be used to automate Windows GUI tasks by simulating user key strokes and mouse actions.

AutoIt is actively maintained, has a broad users base, and supports all versions of Windows.
AutoIt consists of a function library, a Basic-like programming language, and a compiler which allows you to create redistributable .exe scripts.
The recommended editor for AutoIt is a modified version of SciTE, available through the AutoIt web site.
As of April 2010, there is no Eclipse plug-in for AutoIt. The probability of such a plug-in ever being developed is low.
The function library focuses on window, keyboard, mouse, process management.
You can also create simple GUIs with AutoIt (if you must)
AutoIt scripts can call COM objects, providing access to additional functionality. There is a nice tutorial on oleview.exe included in AutoIt's help file, which may be of interest even if you don't need AutoIt as such.
Support for finding and manipulating controls other than by keyboard/mouse simulation and reading contents of GUIs seems missing or limited at best. For these reasons, AutoIt should be considered an automation tool to speed up repetitive tasks, not a full-featured GUI testing tool.

The longer I stay involved in software development business, the more I dislike the idea of inventing new specialized programming languages. Generally, by inventing a new language you are making life harder for everyone out there for two reasons:

established, well-known tools that took thousands of hours of development effort are likely to be useless with your new language
established, well-known libraries that took thousands of hours of development effort are likely to be more difficult to access from your new langugage
your new language will suffer from performance problems solved by thousands of hours of development effort invested into compilers for existing languages

A good reason for a custom programming language is if your users are so unsophisticated and their programming needs so limited that they would be overwhelmed by having to learn an existing scripting language. This might have actually been the argument when the first version of AutoIt was developed, but doesn't seem to be the case any longer.

Another good reason for a custom programming language is if you want to be cut dependencies. For example, write a custom binary compiler, which is just not there for a general-purpose scripting language. This is an advantage that AutoIt's author realized by making it possible to compile and redistribute AutoIt scripts as (small!) .exe files.

Anyway, when it comes to real-world scenarios found in my projects, I thought I might prefer to use AutoIt's automation functionality from a real-world language with a familiar syntax and direct access to tons of third-party libraries. Perl is one such language, hence I spent a little time testing AutoIt-Perl integration.

AutoIt ships with AutoItX, a DLL containing a COM object through which you can script AutoIt itself from within programs written in a language such as Perl. I experimented to see how well it goes. It does work, although there are some caveats. Below I include an example with some notes in comments.

use Win32::OLE;
use Win32::GUI;

# Note 1: we initialize OLE using Single-Threaded Appartment (STA) explicitly,
# as this is what the AutoIt COM object requires. If we don't do it,
# the default init via Win32::GUI::Window->new would initialize using
# Multi-Threaded Appartment (MTA), which would cause AutoIt calls to fail silently.

Win32::OLE->Initialize(Win32::OLE::COINIT_OLEINITIALIZE());

$win = Win32::GUI::Window->new();

# Note 2: we must rely on ShellExecute from Win32::GUI rather than from AutoIt.
# Although there is a ShellExecute function available in AutoIt, it is not exported
# via the COM interface (AutoItX). In general, not everything available in AutoIt
# is exposed via AutoItX (which is bad).

# Note 3: when launching MS Access to open a database in a "trusted location"
# (i.e. macros allowed), then the path to the opened database must match exactly
# the path configured for the "trusted location". In particular, backslashes rather
# than slashes must be used as separators.

# Open MS Access database

$win->ShellExecute(
    "open",
    "msaccess.exe",
    "k:\\workspace\\nca_bo\\nca_bo.mdb",
    "k:\\workspace\\nca_bo",
    10);
    
# Note 4: see ShellExecute API doc for the meaning of constant 10
# in the above invocation.

my $Au3 = Win32::OLE->new("AutoItX3.Control");
# Wait till a pop-up message box appears
$Au3->WinWaitActive("Microsoft Office Access", "vom Web-Portal importieren");
# Simulate Alt-N keystroke to close message box
$Au3->Send("!n");
# Wait till an Access form appears
$Au3->WinWait("Hauptmenü");
# Simulate Alt-E keystroke to push a button on the form
$Au3->Send("!e");
# Wait till a pop-up message box appears
$Au3->WinWaitActive("Microsoft Office Access", "zum Web-Portal exportieren");
# Simulate Alt-N keystroke to close message box
$Au3->Send("!n");

Closing remarks:

To find out about the AutoItX COM interface, consult a .chm help file shipped with AutoIt, but separate from the main AutoIt help.
Lack of proper error reporting is a major risk when using Perl-AutoItX together, and possibly also for AutoIt alone. I got into situations where a command simply did not work, without any apparent error message (but in each case I could find a reason through closer investigation of the expected/actual syntax, Google).
I haven't tested the reliability of transferring data between AutoIt and Perl. Typically, integration between programming languages shows its flaws in the call API when it comes to exchanging more complicated data structures, passing parameters by reference etc.
Depending on the complexity level of the actual scripts, it may be actually tolerable to rely on coarse-grained integration: write scripts using AutoIt syntax and SciTE, then encapsulate them as functions into Perl.
In any case, putting serious functionality into AutoIt scripts that can be just as easily be implemented in a more mainstream language should be avoided when possible. For the particular combination of AutoIt and Perl, the general rule of thumb should be "as little AutoIt as necessary, as much Perl as possible".

Version control in Drupal 6 using the Features module

2010-04-12T02:01:00.005+02:00

Warning: in this article I assume that you understand the concepts of modules and hooks in Drupal, and that you also know the Views module, which is used as an example. It also assume that you are at least somewhat familiar with the general principles of software version control and are looking to apply them to developing Drupal 6 sites. It is also advisable that you have a brief look at the monkey-see-monkey-do screencast about the features module, which currently substitutes comprehensible documentation.

In 1972 programmers realized that efficient collabaration within software projects is difficult without tools that efficiently support exchanging and comparing changes made to source code. In that year the first source code control system was constructed. With the later release of CVS, which remains popular until today, the problem has pretty much been solved in a satisfactory way (though Mr. Linus "Git" Torvalds and others like to disagree).

All traditional version control tools exploit a simple observation about the nature of source code evolution: most of the time, in most well-structured programs, meaningful changes are represented as alterations of consecutive lines of source code. Changes, and their effects, tend to be local. Most programmers also naturally work and think in terms of changing lines of source code within files. Thus, all you need to satisfy most programmers is a system which quickly and conveniently presents which lines have been changed by whom, given two or more points in time. With that ability, you can also package sets of changes into compact files called patches and then apply them to an original version of a system to reliably reproduce someone else's work or store away to track individual contributions (and blame for bugs).

The idea easily generalizes to tools for managing configuration items rather than source code. The term configuration item is rather related to ideas such as "configuration of electrons in an atom" or "configuration of stars", not to be confused with the idea of configuration files. It refers to any artifact which might change over time and changes to which must be propagated between individual developers or systems. If changes to configuration items are also local, it is most straightforward to map them to simple text files and use a traditional version control tool to manage them. The opposite, bad approach, would be to convert simple text files into something that the version control tool cannot effectively deal with (something neither text- nor line-oriented) and thus to produce a classic configuration management problem.

Fast forward to 2009, Drupal 6 (and other Content Mangement Systems, to be fair). It appears that in this new context young people are now making some noise about the SCM trap. Putting configuration items, such as Drupal CCK content types, views, permissions settings - just about anything - into a relational database reliably defeats traditional version control tools. It becomes as cumbersome to distribute and coordinate changes to these items as it used to be for source code changes pre-1972. Yet, luckily for everyone, approaches and tools for solving this age-old problem are also being (re-)invented. The unfortunately named features module for Drupal 6 (SEO, anyone?) provides a workable solution, which I elaborate in the rest of this article.

However. before we go any further, a word of caution regarding terminology is in order. The established term "configuration item" (by Professor W. Tichy, during the 80's) is apparently unfamiliar to or not good enough for inventive young Drupal developers. They like to call configuration items "exportables" or (much worse) "components". This is the terminology found in the original code of the features module, the documentation, and the API. You better get used to it if you want to understand these sources, but I will try to shield you from confusion as much as possible by academically sticking to configuration items in this article. It is intriguing why so many otherwise intelligent and dedicated people choose to pollute the world with lazy rubbish terminology. In case of key projects it is likely to then become perpetrated by hundreds (thousands?) of unsuspecting follower-developers. A worthy topic of research for a psychologist, no doubt. But I digress.

The Drupal features module allows you to

Export configuration items from the database to a set of text files. (Ideally, one file per configuration item should be used to reflect the locality of changes and to reduce the risk of file-level conflicts. Grouping of similar configuration items in one file is also acceptable, and is indeed employed by Drupal's features module.)
Exchange and manage contents of the exported text files using traditional version control tools.

At this point in our design reconstruction, different approaches seem viable. An intuitive one, which I and my colleagues have been successfully using in another (non-Drupal) project for a few years, is to provide a symmetric import operation which loads the (now merged/resolved) configuration items from the file system back into the database. However, the solution used by Drupal's features is even neater, as it eliminates the need to perform manual imports (and thus also the danger of forgetting an import):

Use the exported files as the authoritative (live) version of the represented configuration items...
...unless another, more up-to-date version of those same items exists in the database. In this case, treat the database version as authoritative. That is, database settings override, or take precedence over, file system settings.
Make it possible to re-export the modified configuration items from the database. Consequently, the exported files' content becomes identical to the database, and so again, these files may be treated as authoritative.

In summary, people are supposed to edit configuration items in the database and always re-export them before synchronizing with the version control repository. In this way their changes become available to others and others' changes can be merged in through the file system, while the GUI used for editing may still be implemented in the browser (making it more or less comfortable, but certainly more fool-proof than source code editing).

A specification of which changes are actually exported (i.e. "belong to a feature"), is maintained within a (custom) feature module; not to be confused with Drupal's features module, which is used to generate the former one. A custom feature modules furthermore includes a plain-text version of the actual configuration items, not just references to them. More precisely, a source code representation of those items is kept within the feature module. By the way, note that a feature module is in itself a configuration item represented in the file system and thus readily accessible to version control. However, it is not a configuration item in the sense of possibly having a shadow copy within the database. It's more akin to a normal Drupal module. However, as mentioned, its content is generated by the features module and it also does contain some additional metadata that allows the features module to treat it as its own creation when it comes to re-exporting.

Asking inconvenient questions early is a virtue. Given the astonishing number of Drupal modules, each of which may contribute their own configuration items to a web site by introducing its own database tables, or even worse, extending existing ones, how does the features module find out how to export all those alien configuration items? And how do those alien modules know how to use the disk versions stored within the exported feature module instead of looking for versions in the database? Under which conditions is an exported configuration item reusable across different versions of an originating module? Is it even sufficient to check that the originating module is installed in the right version, or must we ensure that other supporting modules are also available on an item-per-item basis? More generally, is the beta version of the features module ready for use in real projects. Does it introduce any significant risks (such as data loss, lack-of support time bombs), glitches or limitations? To address these issues, in lack of case studies, a closer look under the hood of the features module is necessary.

To see what happens (and where and why things might go wrong), let's walk through a scenario of creating and using a feature which contains just a couple of view definitions:

To begin creating our feature, we fill out the export form located at /admin/build/features/create. Here we see an "Add components" combo box filled with entries such as "Content types", "Permissions", "Views", "Dependencies". In short, what is referred to as components here is really the kinds of configuration items that can be exported as part of a feature, Elsewhere, the developers carelessly use the same bland term component to refer to an individual configuration item, thus committing the ~~modelling shortcut~~ embarassing error of confusing types and instances, and wreaking havoc in minds of their readers. To populate the combo box, the features module invokes hook_features_api.
Interestingly, implementations of this hook are not contained within the contributing modules, but rather within the features module itself. This might be interpreted as a commitment of the features module's maintainers to keep up-to-date with changes in important (core) third-party modules. It might be also interpreted as a non-commitment of the third-party modules' authors to play nice with features. As the features module is rather new, the choice by its original inventors to put the burden of first-time integration on their own shoulders is understandable and smart from the division of labor viewpoint. In the long term, it might represent a weak point of the overall scheme and a stability risk, especially given the currently inadequate amount of examples/documentation for third-party integration.
Anyway, we already see an answer to one of the questions posed above about how the features module knows what to export. It either includes direct support for a particular kind of configuration items, or (preferably) relies on third-party module authors to describe their own kinds of configuration items through an API.
When we select a kind of configuration item in the "Add components" combo box, a list of individual exportable configuration items of that kind below is displayed. The content of this list is generated by hook_features_export_options. In case of views, the features module implements this hook directly to return names of all enabled views. This hook also provides an opportunity for the implementor to influence in which file the exported configuration items will be stored withing the feature module (e.g. in the same or separate file from configuration items of the other kinds). The views-specific implementation opts to store all exported views definitions in a single, separate file.
Suppose we click on a particular view in the list. Through an ahah callback the server is now consulted about further details of the selected configuration item, and the right-side pane of the form is updated with those details. More precisely, hook_features_export is invoked with a list of identifiers of all currently selected configuration items (e.g. view names). An empty output container is also passed in as a parameter. The hook's implementation is supposed to transfer the input list to the output container. At this stage additional items, not present in the original list, might be added. Furthermore, module dependencies of the selected configuration items are determined and recorded in the output container.
The provision of dependency recording gives a (partial) answer to the question of reusability of the exported configuration items in new contexts. The capability to record module dependencies allows the features module to disallow enabling a feature which contains configuration items not supported by available modules (or alternatively, to prompt the user to download the required modules).
The hook implementation may also designate follow-up functions to be called after return to continue the process of filling in the output container. This mechanism is used if the exported configuration items in fact consist of sub-items, which must be handled by other modules. In case of views, there is no need for such a cascade. The implementation simply transfers the requested view names to the output container and records names of modules on which the associated view definitions depend.
When we finally submit the export form, hook_features_export_render is called to query implementing modules for the actual exported source code to be stored within the feature module. The views-specific implementation calls the export method on each view to obtain this source code. Generally, it's entirely up to the implementor how each configuration item is represented in the source code. The views-specific implementation generates source code that populates and returns an array of view definitions.
Besides of the source file which holds all configuration items of a kind, the features module also generates three other files: an .info file, which includes a description of module dependencies, a boilerplate .module file, and a .features.inc file. This last file is of particular interest, as it defines a single hook for each kind of configuration item that was declared using hook_features_api (the same hook used to populate the combo box in first step, remember?). In case of views, the generated hook's name is declared to be views_default_views. This hook is what the views module will later call back to look for any exported views definitions. The hook's code is boilerplate - it delegates to the real function, which returns the array with configuration items (view definitions, in this case).
Having generated all those files, the features module packs them together and offers the resulting feature module for download and installation. The installation (enabling of features) occurs through the /admin/build/features page, rather than the usual modules page.

Thus far, we have roughly examined what happens during (re-)creating or (re-)defining a feature. Now it's time to look what happens when a feature is enabled. As it turns out, not so much. The function features_install_modules is called with a list of all feature modules that you have chosen to enable. This function silently ignores all already enabled feature modules and only processes the additional ones. For each new feature module, it determines a transitive closure of dependencies and enables all those required modules if need be. The new feature module itself is also enabled.

The next step is to examine how modules that own and export configuration items utilize their exported representations. More particularly, how does the views module determine whether to use a view definition stored in the database or the one contributed by a feature module through the file system? The solution is rather simple and implemented with little support from the features module. As already mentioned, a hook (views_default_views) hook is generated during the export. The views module calls this hook to find out which views are represented as exported configuration items (aka "default views"). Depending on whether or not a databased version of a view is also available, it is returned from views_get_view instead of the default one from the file system. When a view is updated, a version of it is created in the database, possibly using the default view as a prototype. No surprises.

Finally, let's tackle the remaining few operations that can be performed on an enabled feature: reverting, rebuilding (aka recreating or re-exporting), and disabling:

Reverting a feature means restoring all configuration items to their original exported state. Any changes made to the database versions of the configuration items are undone. The implementation within the features module simply calls the hook feature_revert and lets the modules that own the configuration items do the job. In case of views, the database version of each view included in the feature is deleted, thus giving precedence back to the default view. However, according to my tests, the views module seems to remember whether a view has a database version equal to the file system version originally, and in such case does not delete the database version during feature_revert, though you can delete it using the Revert action in the views GUI.
Recreating a feature means re-exporting the current versions of all included configuration items. The only difference from the original export is that now you don't have to choose which item to include in the feature (although you are given this opportunity). The process gives you a new .tgz feature module with which you should overwrite the installed module (unless it contains external changes, in which case you should merge both using your version control tool).
Disabling a feature is quite simple - it just disables the feature module, making the database versions of the configuration items all that remains active. A word of warning: if after enabling the feature you remove the database versions of the configuration items and subsequently disable the feature, you will effectively remove these configuration items from your site. For example, if you revert a view to only rely on the file system version, and disable the enclosing feature, the view will be gone (until you re-enable the feature, that is). This is to be expected. Also, if you choose to recreate such a disabled feature, the set of exported configuration items will not match the original one - in particular, the views which don't have versions in the database, will not be re-exported, unless you explicitly add them again to the export set.

Having provided (admittedly roundabout) answers to the questions posed in the beginning of the article, let's turn in conclusion to the bigger issue: is the features module safe to use today?

Obviously, an answer would benefit from more real-world experience to better estimate the frequency and severity of bugs. Still, even after a short examination of the overall design (and some peeks at the source code as well), it appears to me that the core concept of features is solid and ready for prime time. The unfortunate terminology and confusing documentation are flaws, but they are not insurmountable and certainly not show-stoppers, judging from the quality of other popular Drupal modules. I shall give the features module a serious try in my project to alleviate the pressing need for conveniently exchanging views and content types definitions between developers and tracking changes. One apparent weakness is that at present some important kind of configuration items might not be supported (e.g. localized strings). However, the features module can be augmented with other export-to-file-system approaches where they are available (.po files), and with background information included in this study, integration with new modules should also be relatively easy.

Having written this article, I actually started using the Features module in one project to keep the development, test and production sites in sync. The following updates document some discovered glitches and "lessons learned".

Update 2010-04-18, uploading updated CCK content types to production

I discovered the following caveat when using features to synchronize CCK content types from development to production: when you add a new field to a CCK content type, recreate the feature, then upload the updated feature module to a production site, the new field is not added to the content type immediately (neither is it added to the content type's database table). Instead, the feature appears marked as "needs review" and you have to "revert" it manually in order to make changes active. I would expect the "needs review" state to only occur if there have been independent changes to the feature's configuration items at the production site since the last upload, that is, when there is room for conflict, so this behavior is surprising.

Update 2010-05-12, added CCK fields not included in feature automatically

Suppose you have an existing feature which contains a content type. You add a new field to the content type in your database. What state would you expect to be reported for the feature then? Well, I would like it to be treated the same way as the case of a view to which I added a new field - the feature should show up as "overridden", and recreating the feature should export the new CCK field. Yet it doesn't. Apparently you have to add the new field to feature.info manually for this configuration (sub)item to be included in the feature at all.

I think that the reason for the current state of affairs might be the developers' speculation that a single content type may be governed by more than one feature (as well as the user herself). However, the same could be said about fields of a view. In any case, the current behavior is inconsistent.

Update 2010-05-12, modules added as dependencies not enabled automatically

Suppose you enable a module in your development site and add it as dependency to a feature. You then upload your feature to a production site which has the module in disabled state. What would you expect to happen when you revert the feature to "default" state? I would expect the module to become enabled (or some kind of error message to be displayed, had the module been unavailable). However, what happens instead is that the module stays disabled and - worse - can no longer be enabled manually because it is "locked" through the feature. You have to remmber to enable the module manually before uploading the updated feature which depends on it. This is exactly the kind of thing which you should not have to remember when relying on a configuration management tool.

Drupal 6: Translating user-defined (dynamic, variable) strings

2010-03-04T18:49:00.008+01:00

There are times during Drupal module development when you want to translate a string which is defined by the user and read from the database. You are not supposed to apply the t() function in this case because it should only be used for static strings with variable (non-locale-specific) replacements. (Break this rule if you want to confuse the localization tools reponsible for extracting strings from code.)

As an example of the problem, consider Drupal's core contact.module, which at the time of writing is (regrettably) not fully localized. It allows the user to define categories, whose names are then displayed in the GUI:

Original:

    $categories[$category->cid] = $category->category;

Wrong solution:

    $categories[$category->cid] = t($category->category);

(At least partially) Correct solution:

    $categories[$category->cid] = i18nstrings_ts(
       'contact:category:' . $category->cid . ':name',
       $category->category, NULL, TRUE);

The last solution (relying on i18n.module API) is good. You will be able to find and translate the strings using the localization UI. It has a cosmetic flaw that the localization UI is not really aware of the new 'contact' textgroup, so the group name will not be displayed at all. If you want a full-fledged solution, follow instructions on making your custom data translatable and study i18ncontent.module or i18views.module for further details.

Solution for $_POST empty after drupal_http_request

2010-03-03T14:49:00.004+01:00

The function drupal_http_request can be used to POST data to another Drupal script. However, what the documentation does not mention is that when you invoke it without the Content-Type header, the receiving script might not see any of the posted content.

Wrong:

  $result = drupal_http_request(
    url('some/url', array(absolute => TRUE)),
    array(),
    'POST',
    'param1=value1¶m2=value2');

Correct:

  $result = drupal_http_request(
    url('some/url', array(absolute => TRUE)),
    array('Content-Type' => 'application/x-www-form-urlencoded'),
    'POST',
    'param1=value1¶m2=value2');

Drupal 6: Adding form fields dynamically

2010-02-17T15:05:00.005+01:00

There are times when you want to understand how something works, but there are plent more times when you simply want to get things done. When it came to dynamically extending Drupal 6 forms per Ajax/AHAH, I found some half-baked explanations of the "ahah voodoo" (aka implementation details) on the web, references to poll.module as the place which contains such frightening "voodoo", but no simple examples which would provide the usual delights of copy-paste reuse.

The AHAH idea is very simple - call back the server to generate a HTML snippet with a few more form fields to be inserted and subsequently submitted by client. Alas, implementing even such a basic scenario in Drupal 6 is complicated. To improve the matters a little, I implemented a Simple AHAH API module (download link). Example client code for an order form with dynamically added order items is shown below. It is almost self-explanatory (for further instructions, see included README.txt). Overall, it should reduce the implementation time for the most usual case (dynamically adding form fields) from several tiresome hours of debugging to a few minutes of thoughtless fill-in-the-blanks.

function nca_ahah_init() {
  module_load_include('inc', 'nca_ahah', 'nca_ahah');
}

function nca_ahah_menu() {
  $items['nca_ahah/nca_ahah_order_item'] = array(
    'title' => 'Javascript Order Item Form',
    'page callback' => 'nca_ahah_callback',
    'page arguments' => array('nca_ahah_order_item'),
    'access arguments' => array('access content'),
    'type' => MENU_CALLBACK,
  );

  $items['nca_ahah/order'] = array(
    'title' => t('Order items example'),    
    'page callback' => 'drupal_get_form',
    'page arguments' => array('nca_ahah_order_form'),
    'access arguments' => array('access content'),
    'type' => MENU_CALLBACK,
  );
  
  return $items;
}

function nca_ahah_theme() {
  return array(
    'nca_ahah_order_item' => array(
      'arguments' => array('form' => NULL),
    ),
  );
}

function nca_ahah_order_form($form_state) {
  $form = array();
  
  nca_ahah_wrapper($form, $form_state, 'nca_ahah_order_item');

  $form['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Proceed to checkout'),
    '#weight' => 2,
    );

  // Avoid submitting the form to AHAH handler on checkout. 
  $form['#action'] = url('nca_ahah/order');
  
  return $form;
}

function nca_ahah_order_form_submit($form, &$form_state) {
  if ($form_state['clicked_button']['#id'] == 'edit-submit') {
    // Real submit button was clicked.
    dpm($form_state['values']);
  }
  else {
    // "Add more" button was clicked.
  }
}

function nca_ahah_order_form_validate($form_id, &$form_state) { 
  if ($form_state['clicked_button']['#id'] == 'edit-submit') {
    // Real submit button was clicked.
  }
  else {
    // "Add more" button was clicked.
  }
}

function nca_ahah_order_item_form(&$item_form, $i, $item) {
  $item_form['item'] = array(
    '#title' => t('Item name'),
    '#type' => 'select',
    '#options' => array(0 => '', 1 => 'First Item', 2 => 'Second Item', 3 => 'Third Item'),
    '#default_value' => isset($item['item']) ? $item['item'] : 0,
    '#parents' => array('nca_ahah_order_item', $i, 'item'), // unfortunately needed
    );

  $item_form['qty'] = array(
    '#title' => t('Quantity'),
    '#type' => 'textfield',
    '#default_value' => isset($item['qty']) ? $item['qty'] : 0,
    '#size' => 5,
    '#maxlength' => 7,
    '#parents' => array('nca_ahah_order_item', $i, 'qty'), // unfortunately needed
  );
}

function theme_nca_ahah_order_item($item_form) {
  // You can do more fancy layout/HTML formatting here
  return drupal_render($item_form);
}

Drupal 6: Debugging form submissions

2010-02-13T00:02:00.005+01:00

This post contains a few notes taken from a debugging session for a problem of the kind "submitted form is being returned to the sender without any error message" in a Drupal 6 installation. Furthermore, it contains some tips on debugging the "form values disappear into the void" symptom. These are surprisingly difficult issues to debug if you don't know where to start, as Drupal's control flow is notoriously convoluted due to indirection caused by hooks (as a matter of fact, quite a few software frameworks pay the same price for their flexibility) and because of generous use of recursion.

Check if form_set_error is called (with a non-empty argument) by any module during processing.
Instrument or break in function form_execute_handlers in form.inc. This is from where the validate and submit functions contributed by various modules are invoked from.
If the form validates, but an expected submit handler is not in the list, it may be due to the form id not matching the function name. If you cannot change either, you can still bring them together in the form_alter hook by appending to #submit, like so:
```
$form['#submit'][] = 'user_register_submit';
```
If the form values in $form_state seem to be wrong or missing, debug the $form['#value'] assignments in _form_builder_handle_input_element (to conditionally break on a particular element, you can evaluate $form['#name']). In particular, for multistep forms pay attention to the #access attribute, which controls whether the values are populated from the form cache or from the POST request.

Solution for MSIE 7 only applies style on refresh

2010-01-29T15:17:00.003+01:00

Problem: Internet Explorer 7 only renders some pages correctly after they are reloaded. The originally loaded page seems to have some of the stylesheet applied to it (e.g. colors), but not everthing (e.g. font). Furthermore, the View Source menu option does not work - MSIE appears to be stuck rendering the page. It happens to some pages of a web site, while others are rendered correctly for no apparent reason. Everything works perfectly in Firefox (or even in MSIE 6).

Solution: This particular problem has been caused by a link to the stylesheet incorrectly placed inside of the BODY tag instead of inside of the HEAD tag. Simply moving the LINK tag in page's source fixed it.

LVM2 snapshots: device-mapper: create ioctl failed: Device or resource busy / Failed to suspend origin

2010-01-24T18:05:00.003+01:00

If you are getting the error "device-mapper: create ioctl failed: Device or resource busy" when using the lvcreate command to set up an LVM2 snapshot, and also seeing an EBUSY on a second call to DM_DEV_CREATE ioctl when you strace the command, and finally seeing "unknown filesystem type 'DM_snapshot_cow'" when you try to mount the snapshot, then here is a clue of what may be wrong and a possible workaround:

I ran into this problem when I set up the origin device (i.e. the one of which the snapshot should be taken) manually using dmsetup (after having it manually removed - to be able to detach the loop device under it). However, as it turned out, the proper way to set up such a device is through the command vgchange -ay.

To find out whether you're having the same issue and to reach a clean initial state, first remove the offending device(s) using dmsetup (you may have to remove any partially set up logical volumes with lvremove before that). You should begin with a state in which vgdisplay -v displays your origin volume as inactive/unavailable and dmsetup ls displays no devices. After that, invoke vgchange -ay. The origin device status should now change to available and dmsetup ls should be showing the corresponding device. You can also mount it at this stage. Finally, you can (hopefully) create a snapshot without running into the above described problem.

Drupal 6: Master/detail tabbed navigation

2010-01-04T18:07:00.006+01:00

Master/detail navigation is a common scenario in graphical user interfaces for databases. After selecting a master record, the user is given the choice to view all detail records of a particular type that are in some way related to the master record. A natural design choice is to offer tabbed navigation to support this scenario. The first tab contains data from the master record, while subsequent tabs represent different types of associated detail records in tabular format.

The rest of this post explains how to implement the above scenario in Drupal 6. To be more concrete, let's assume that we have a CCK content type Tournament, which is associated by node reference with a CCK content type Result (a tournament contains multiple game results). We also have two views, Tournaments and Results, which list all nodes of the respective type using a table display and furthermore contain exposed filters for all fields - including the tournament title field. The intended final GUI is shown in the following two screenshots (click to enlarge):

The steps to implement the tabbed master/detail navigation are outlined below. Note that the solution does not use the official Views 2 API (which is still quite undocumented at the time of writing). It is based on trial-and-error experimentation - use at your own risk.

Add a Tournament argument to the Results view definition. The argument should be configured with "Display all values if not present", and employ a validator for the node type Tournament. When set up correctly, the results of a particular tournament may be displayed by specifying the view's path followed by the tournament node ID. For example, invoking the view through path /results/12345 would only display results of tournament 12345.
Note, however, that the Tournament filter and Tournament field, though redundant in the argument-based view, still appear. We will deal with them a bit later.

Add a "local task" type menu item to create a tab for the Tournament node type. We limit the appearance of the tab to Tournament nodes by specifying a custom access callback.

function bbo_scoring_hook_menu() {
  // ...
  $items['node/%node/bbo_scoring/results'] = array(
    'title' => t('Results'),
    'page callback' => 'bbo_scoring_tournament_results_page',
    'page arguments' => array(1),
    'access callback' => 'bbo_scoring_tournament_results_access',
    'access arguments' => array(1),
    'type' => MENU_LOCAL_TASK,
    'weight' => 100,
  );
  // ...
}

function bbo_scoring_tournament_results_access($node) {
  return $node->type == 'tournament';
}

Implement the page function so that it outputs the embedded Results view. By the way, we hide the redundant exposed filter for tournament and the tournament field on this page. Regrettably, for some reason this trick does not work in view pre-hooks, which would seem a more universal solution.

function bbo_scoring_tournament_results_page($node) {
  $view = views_get_view('results');
  $view->display['default']->display_options['filters']['field_tournament_nid']['exposed'] = FALSE;
  unset($view->display['default']->display_options['fields']['field_tournament_nid']);
  $display_id = 'page_1';
  if (!$view || !$view->access($display_id)) {
    return '';
  }
  return $view->preview($display_id, array($node->nid));
}

Alter the filter form's action to lead back to the tab page. Without this step, the default URL of the view would be displayed after applying the filter.

function bbo_scoring_form_views_exposed_form_alter(&$form, $form_state) {
  if (arg(0) == 'node' && arg(2) == 'bbo_scoring' && arg(3) == 'results') {
    $form['#action'] = '/node/' . arg(1) . '/bbo_scoring/results';
  }
}

Drupal CCK - concepts and database tables

2009-09-30T15:38:00.006+02:00

What follows is a brief summary of a nice (but verbose) introduction to the Drupal CCK module, which can be found at What is the Content Construction Kit? A View from the Database:

CCK allows the site administrator to define custom content types (aka node types) and fields. Content types define data structures and furthermore contain user input/validation rules needed to generate UI forms for populating those structures. Content types are like 'tables' or 'relations' within a relational database, while fields are like 'attributes'.
For each user-defined content type, CCK adds a table named content_type_type name
Different field types correspond to different data types. In particular, a node reference type is available for implementing 'foreign keys'.
For each field, CCK stores type-specific metadata in tables node_field (global data for each field, e.g. storage details) and node_field_instance (content type specific, e.g. contextual presentation details). The same field may be reused by multiple content types.
For fields of type 'single' (1:1 association between nodes and field values) that are not shared between content types, CCK stores field values and in columns field_field name_value of the respective content type table. Additional columns may also be stored there for the field (e.g. field format).
For fields of type 'multiple' (1:n association between nodes and field values) or shared between content types, CCK adds a table named content_field_field name



Profiling Drupal performance
2009-08-31T15:08:00.008+02:00
Example problem: Rendering the Administer page is very slow - it takes upwards of 20 seconds on a development machine without any load.
Solution
Browsing Drupal forums, you will find plenty of theories and more or less helpful tips. However, it is usually more productive to work from first principles, that is, according to the general approach described below.
Begin by enabling query and page build time logging in Devel module's configuration. For extra certainty, also enable query logging in the MySQL server. If the problematic page's queries execute fast, the problem must lie somewhere in the PHP code. PHP code can be easily profiled using the XDebug profiler. After installing the XDebug extension, simply enable the profiler by adding the following in xdebug.ini:
xdebug.profiler_output_dir=/tmp
xdebug.profiler_enable=1

(Beware of doing it in production environment; profiling tends to be resource-intensive.)
Restart Apache, reload the problematic page and wait until it finishes rendering. You will see that the profile /tmp/cachegrind.out.nnnn with execution time/count measurements was created. The profiler can now be disabled unless more data is required.
Inspect the profile by opening it with KCachegrind:
Click to enlarge
In the above screenshot KCachegrind pinpoints the function drupal_get_schema_versions as consuming almost 24 of the total 27 seconds execution time. A quick Google search reveals that this performance problem has been observed and solved for Drupal 7. After a quick-and-dirty test that disabling that function indeed fixes the performance problem, you can pursue the issue further, for example by adapting the most recently submitted patch to your installation. (Take into account the increased cost for future upgrades if you follow that route.)



Understanding Node Access Control in Drupal
2009-08-29T15:58:00.016+02:00
This post contains the (pseudo-code) algorithm used by Drupal 5 to determine whether or not a given user may view/edit/delete a given node. The description is based on the Pro Drupal Development book (p. 104), while later parts about access control modules are based on source code. It assumes familiarity with Drupal's concepts of nodes, modules, and hooks.
Node access control algorithm in Drupal core
Columns of node_access:nid
gid
realm
grant_view
grant_update
grant_delete
First, the user may be granted access based on the information obtained from modules implementing the hook_access callback. If no information is obtained that way, then per-node permissions stored in the node_access table are consulted. This table's contents are maintained incrementally by modules implementing the hook_node_access_records hook, which is invoked whenever a node is saved. Each row in that table has the following meaning: "If the user owns the grant $gid, then she may view/edit/delete node $nid". It is sufficient that the user matches one such row to be allowed access. Accordingly, it is not possible for one module to deny access allowed by another module using this mechanism (unless it implements hook_access to have the first say and skip the grants-based access altogether - which is unfortunately impossible for CCK node types without patching). The set of grants owned by the current user is determined by modules implementing the hook_node_grants callback. This set is tied to the user and its content is independent of any particular node.
Note that the table node_access is shared by multiple independent modules. To avoid collisions, each module's grant IDs are only unique within its namespace, called a realm. Realms play no further role in access control.
The "intrinsic" meaning of a grant ID depends on the module which contributed it and is not relevant for the generic access control module. For example, the Taxonomy Access Control Lite module uses term IDs as grant IDs ("the user has such-and-such terms associated with her; the node can be viewed/updated/deleted by any user who has such-and-such terms associated with her"), while the Workflow module uses role IDs as grant IDs ("the user has such-and-such roles associated with her; the node can be [in its current workflow state] viewed/updated/deleted by any user who has such-and-such roles associated with her").
The incremental updating of the node_access table may (and does) lead to inconsistency if updating a node fails in a non-atomic fashion (that is, without rolling back the database updates that have been already triggered by the update). As a practical example, consider a change of a node's workflow state. This triggers a node update, but may also trigger a user-defined action associated with the state transition. If the action fails (e.g. due to an error in the PHP code), then execution is aborted even before hook_node_access_records has done its job. Drupal 5 lets the administrator rebuild node_access upon request. This is semantically equivalent to clearing the table and saving each node once. Of course, it doesn't help with detecting why the data corruption has occured in the first place, so it should be considered a half-measure.
Here is the simplified access control algorithm (the real implementation can be found in function node_access in node.module):
if ($op == 'update' && !has_access_to_input_format($user))
{
    return 0; // access denied
}

if (has_administer_nodes_permission($user))
{
    return 1; // access granted
}

if (!has_access_content_permission($user))
{
    return 0;
}

if (is_published($node))
{
    $result = hook_access($user, $op, $node); // for node type

    if (!is_null($result))
    {
        return $result;
    }

    // get user's grants
    $grants = hook_node_access_grants($user);

    // check whether the user may view/edit/delete node based on her grants
    return is_permission_granted($grants, $op, $node);
}
else
{
    return is_node_author($user, $node);
}

The ACL module
The ACL module provides a slightly higher-level API for maintaining user-node permissions. It is intended to be used by client modules that need to maintain (possibly directly editable) sets of individual users and grant node access permissions based on whether a user is a member of a particular set. Like any access control module in Drupal, it relies on the node_access mechanism described earlier, and can be thought as an "adapter" or translation layer above it. It offers the following functionality:

Provides an API for other (client) modules to maintain so-called ACLs. An ACL is simply a named set of users. The acl_id designating an ACL is automatically generated, while the name of the ACL - and its meaning - can be chosen freely by a client module. The ACL definitions are stored in tables acl and acl_user.
Provides an API to associate ACLs with nodes and view/update/delete permissions. For example, a client module might define that users on a particular ACL may update a particular node. This information is stored in table acl_node.
Translates contents of the acl and acl_user tables into grant IDs and contents of the acl_node table into node_access records. Here, the string 'acl' is used as the realm name and acl_ids are used as grant IDs.
Contains some functions that man be used by client modules to produce UI forms for manual editing of the ACLs.

Despite being a small API module intended for other clients, the ACL module is poorly documented and mainly used by other modules of the same maintainer, one of which is discussed next.
The Forum Access module
The Forum Access module, maintained by the same person as the ACL module, is one of its few known clients. Let us briefly examine the relationship between these two modules.
The simplistic assumption that Forum Access "builds upon" ACL is proved wrong by the fact that it maintains its own 'forum_access' realm in the node_access table and does it without going through the ACL module. User role IDs are used as grant IDs in this realm and the role-forum permissions are maintained privately in the forum_access table (again, not an ACL). This mechanism is used to enforce role-based access control.
However, in addition to that the Forum Access module maintains one ACL per forum, which is named using the forum's ID (which, by the way, happens to be a taxonomy term ID). Users on that ACL are the "forum moderators", with full view/update/delete permissions on all forum nodes. Thus, forum moderation privileges may be granted regardless of the user's system-wide role.



Solution for "Join expression not supported", can't open query
2009-08-05T17:24:00.003+02:00
Problem: An attempt to open a query in Microsoft Access 2007 results in the "Join expression not supported" error. You can't see the query results nor edit the query to correct the alleged problem because the query view closes immediately. However, you can execute the query from VBA and even export its results to Excel.
Solution:
Debug.Print CurrentDb().QueryDefs("YourQueryName").SQL
in the VBA editor will let you see the (incorrect) query definition. You can set the same SQL property to a new string to override the query definition. However, due to a bug in MS Access in handling JOINs the query might stay unopenable despite attempts to repair:
This problem occurs when a query contains a JOIN expression which links two tables using more than a single column in each of them. For example:
SELECT a.* FROM a LEFT JOIN b ON (a.col1=b.col1 AND b.col2='value')

Access tends to automatically and incorrectly strip the parentheses from the join expression, which leads to the error message mentioned earlier.



Solution for Selenium RC Firefox startup crash
2009-07-21T20:49:00.010+02:00
Problem: A Selenium RC server under Linux starts a Firefox 3 (or 3.5) process, which then crashes immediately. A useless error dialog mentioning Gnome bug-buddy appears, the test script hangs. When started manually, the same version of Firefox works without flaw (including the Selenium IDE add-on).
Solution: (note: this was written for Selenium RC 1.0.1; you may wish to try the most current version (eg. 1.0.3) before troubleshooting)

Check your ~/mozilla/.plugins and remove plug-ins (symlinks) that you don't need. In my case, the Java 1.4.2 Plug-in caused the crash.
If the above doesn't help, gathering more information related to the crash may provide a clue, as described in the following steps.
Write a little wrapper for the firefox script. Save it in /usr/local/firefox/firefoxg or somewhere else:#!/bin/sh
exec /usr/local/firefox/firefox -g $*
 If you start Firefox using this wrapper, you will notice that instead of the browser a debugger opens, in my case, the ddd debugger frontend. You can then launch the executable by typing run in the debugger console. In case of a crash (segmentation fault), you will see a stack trace which may contain interesting clues.
Update your Selenium RC test script to reference the wrapper, e.g. browser => "*firefox /usr/local/firefox/firefoxg".
In the shell where you launch the Selenium RC server, set the environment variable GNOME_DISABLE_CRASH_DIALOG=1, so that Gnome bug-buddy doesn't bother you on crash.
Go ahead and start the test script. The debugger will open. Actually, two instances of the debugger will open, the second one only after the first one is terminated. Type run in the debugger console to continue execution in each case. In my environment the first instance just crashed gdb. However, the second instance produced a backtrace which helped to put blame on the Java plug-in as shown below:




Grep through a MySQL database
2009-05-29T22:14:00.006+02:00
Looking for a particular string in a set of locations is a common theme when debugging or reverse-engineering software. One of the interesting search locations might be a MySQL database. dbgrep.pl is a little Perl script that will select all data from all (or just the specified) tables and then attempt to find a string (or regular expression) match in any field. Needless to say, it is rather resource-intensive, but still fast enough to sift through moderately sized development databases (at a processing speed of 3-5 MB/s).
You can download dbgrep.pl from my web site.
This is dbgrep.pl, version 1.0, which scans for strings in a MySQL database.
Copyright (c) 2009 Jan Ploski (plosquare.com)
This software is licensed to you under Artistic License 2.0,
which is published at http://www.perlfoundation.org/artistic_license_2_0

Usage: dbgrep.pl [-u <db user>] [-h <db hostname>]
                 [-p] [-x] [-a] [-v] [-q]
                 <expr> <db name> [<table name> ...]
Switches: 
       -p a password is needed for authorization, read from console
       -x interpret <expr> as a Perl regular expression rather than string
       -a grep through all fields, not just text/(var)char/blob fields
       -v print matching values, not just table/field/primary key locations
       -q only print names of matching tables and fields

Some invocation examples:
./dbgrep.pl -u dbuser -h localhost -p somestring dbname
./dbgrep.pl -u root -xaq '^123' dbname



Solution for invisible "Save As" dialog in Access/Excel
2009-05-20T22:14:00.005+02:00
Problem: While trying to programmatically export data from an MS Access database to Excel displaying Excel's "Save As" dialog using code like XL.Dialogs(xlDialogSaveAs).Show does not work correctly. This call might work in the VBA debug console, but in a production database the dialog (e.g. triggered by an event) would not appear on screen, although it would show up as process in task manager and furthermore block (freeze) the UI of your Access application. I originally observed this issue in Access 2007 and Windows Vista. At first I suspected threading or parent dialog's modality as causes, but that turned out wrong.
Solution: A hint for solving this problem comes from an observation that it ceases to occur once the Visual Basic editor has been opened. Obviously, this is not going to happen in a production database, but it can be emulated for the purpose of working around the described problem. Just add the following (pointless looking) code to your database's initialization for the later dialog invocation to respond as expected:
    Dim ref As Reference
    For Each ref In Application.References
        If ref.Name = "" Then
            ' nothing needs to be done, just loop
        End If
    Next



Speed up iteration over options of a SELECT element in MSIE
2009-05-13T13:23:00.005+02:00
Problem: An application opens a new browser window in which the user makes some choice. That choice is used to select an OPTION within a SELECT element (combo box, list) in another browser window. The target SELECT contains many (thousands) of values. Because options are not sorted by value, finding the correct one using JavaScript requires a linear loop. This loop takes a very long time (8 seconds), especially in Internet Explorer (version 6 and 7).
Solution: Avoid inter-frame access to the options list of a SELECT element. Instead, define the resource hungry loop within the same frame where the SELECT element is contained. This brings the execution time into the milliseconds range.
I suspect that inter-frame communication in JavaScript triggers some sort of security/access control checks within the browser, accounting for the slowdowns.



Migrating Eclipse update sites to P2
2009-05-07T22:43:00.002+02:00
This article describes how to upgrade an Eclipse update site from the classic, pre-P2 layout (site.xml together with plugins and features directories) to the "new and improved" (read: unnecessarily complicated) P2 layout. Considering that P2, the new update manager introduced in Eclipse 3.4, is backwards-compatible with the old layout of update sites (more or less), why bother at all? In my experience, browsing update sites with many plug-ins and features used to be somewhat slow before, but with P2 it has become excruciatingly slow. Fortunately, P2 also supports a new update site format designed to alleviate this problem, that is, to speed up update site browsing. However, as far as I know, no comprehensive documentation of the upgrade path for those of us still using the classic update site layout exists. More confusingly, the article Update Site Optimization (coming from the source?) only tells half of the story. As I found out, following the somewhat outdated instructions listed in there won't leave you with a working update site. Information from different sources had to be pieced together. Hopefully, no longer, if you read on.
Classic update site vs. P2 update site
Let's start by comparing the classic update site layout with the newer layout tailored for P2. I suppose you have a working update site organized in the classic way and want to switch to the new P2 one for reasons mentioned above:


Classic update site P2 update site

update site = a world-accessible directory on a web server same as before

update site contains site.xml, which contains a list of versioned features installable from this site
update site contains content.jar and artifacts.jar, which together supersede site.xml


subdirectory features contains one JAR file per versioned feature, referenced from site.xml
same as before, but the references now originate from content.jar and artifacts.jar


subdirectory plugins contains one JAR file per versioned plug-in, referenced from feature.xml files contained in feature JARs
subdirectory plugins contains one .jar.pack.gz file per versioned plug-in, referenced from feature.xml as before, but also from content.jar and artifacts.jar


 
update site (optionally?) contains digest.zip - see description below


While the syntax of feature.xml and site.xml is easy to understand and these files are easy to generate/process by your own tools if need be, the new content.jar and artifacts.jar leave no such hopes. You are advised to treat them as P2's private mess (found inside: fat, obscure XML documents). As you will see next, both these JAR files can and should be generated from a slightly modified version of site.xml.
The plug-in .jar.pack.gz files are also generated - each from the original plug-in JAR found of the classic update site. They are essentially JARs recursively compressed with the pack200 tool first introduced in Java 1.5.
The old site.xml and old plug-in JAR files are not ever accessed by the P2 update manager, but if you wish to stay backwards-compatible, you should keep them around in your update site as well.
If you read the Update Site Optimization article, you might be wondering whether a "digest" file (digest.zip) is also needed, where to put it, and what for. I have never observed P2 trying to access this file and suspect that it has been superseded by the content.jar and artifacts.jar duo (which the original article fails to mention). However, it is worth noting that the official Eclipse Ganymede update site does contain a digest.zip in the update site directory and an attribute digestURL="http://download.eclipse.org/releases/ganymede/" on the site element in site.xml (the value of this attribute points to the update site, i.e. location of digest.zip). We'll see next how to generate digest.zip just in case.
How to upgrade to a P2 update site
In essence, the following steps are required:

Add a new attribute pack200="true" to the site element in site.xml.
Add a new attribute digestURL="http://your/update/site/url/" to the site element in site.xml.
Ensure that each of your feature JARs in the features directory contains a feature.properties file (which may be empty). (Here is why.)
Generate a .jar.pack.gz file from each plug-in JAR file in the plugins subdirectory.
Generate digest.zip based on the classic update site (including site.xml).
Generate content.jar and artifacts.jar based on the classic update site (including site.xml).

Steps 1, 2, 3 do not require any sophistication and thus won't be further elaborated. Steps 4 and 5 are (surprisingly) intertwined, as described below. Step 6 is also described in the last section.
How to generate .jar.pack.gz files (Step 4) and digest.zip (Step 5)
The generating .jar.pack.gz files step actually consists of two parts:

For each JAR file in question, "condition" or "repack" the JAR file to prepare it for the second part.
Generate .jar.pack.gz files from a set of conditioned JAR files. Also generate digest.zip

In order to condition (repack) a JAR file, run:
$JAVA_HOME/bin/java \
    -jar $launcher \
    -application org.eclipse.update.core.siteOptimizer \
    -jarProcessor -verbose -processAll -repack -outputDir $output_dir/plugins \
    $input_jar_file

The above invocation contains some variables, to be replaced as follows:

JAVA_HOME path to where Java 1.5 (or newer) is installed
launcher path to plugins/org.eclipse.equinox.launcher_1.0.101.R34x_v20080819.jar from your Eclipse 3.4 (or newer) installation (the version number in the JAR file name may vary)
output_dir path to where the conditioned JAR file should be written - the plugins directory of the upgraded update site
input_file path to the input plug-in JAR file

Sadly, you will have to run the above tool for each JAR file individually. Observe that
the conditioned JARs contain META-INF/eclipse.inf, absent in unconditioned ones.
Finally, copy site.xml and the features subdirectory of your original
update site to $output_dir. This completes part 1.
For part 2, run the following command. Unlike part 1, this processes all the conditioned JARs
it can find through site.xml:
$JAVA_HOME/bin/java \
    -jar $launcher \
    -application org.eclipse.update.core.siteOptimizer \
    -digestBuilder \
    -digestOutputDir=$output_dir \
    -siteXML=$output_dir/site.xml \\
    -jarProcessor -pack -outputDir $output_dir $output_dir

You should now have $output_dir/plugins full of conditioned JARs and a corresponding .pack.jar.gz for each of them. You should also have $output_dir/digest.zip. If not, maybe you forgot to take care of feature.properties in Step 3 mentioned earlier.
How to generate content.jar and artifacts.jar (Step 6)
Here is how (set a human readable $project_name first):
$JAVA_HOME/bin/java -jar $launcher \
    -application org.eclipse.equinox.p2.metadata.generator.EclipseGenerator \
    -updateSite ${output_dir}/ \
    -site file:${output_dir}/site.xml \
    -metadataRepository file:${output_dir}/ \
    -metadataRepositoryName "${project_name} Update Site" \
    -artifactRepository file:${output_dir}/ \
    -artifactRepositoryName "${project_name} Artifacts" \
    -compress \
    -reusePack200Files \
    -noDefaultIUs \
    -vmargs -Xmx256M

The final result
Here is an example P2-upgraded, backwards compatible update site with 1 feature version:

.
|-- site.xml
|-- artifacts.jar
|-- content.jar
|-- digest.zip
|-- features
|   `-- org.epic.feature.main_0.6.34.jar
`-- plugins
    |-- org.epic.debug_0.6.27.jar
    |-- org.epic.debug_0.6.27.jar.pack.gz
    |-- org.epic.doc_0.6.2.jar
    |-- org.epic.doc_0.6.2.jar.pack.gz
    |-- org.epic.lib_0.6.1.jar
    |-- org.epic.lib_0.6.1.jar.pack.gz
    |-- org.epic.perleditor_0.6.23.jar
    |-- org.epic.perleditor_0.6.23.jar.pack.gz
    |-- org.epic.regexp_0.6.1.jar
    |-- org.epic.regexp_0.6.1.jar.pack.gz
    |-- org.epic.source_0.6.34.jar
    `-- org.epic.source_0.6.34.jar.pack.gz

site.xml contains the two new attributes, pack200 and digestURL.
P2 will access content.jar, artifacts.jar, org.epic.feature.main_0.6.34.jar and the plug-in  .jar.pack.gz files.
The classic update manager will access site.xml, org.epic.feature.main_0.6.34.jar and the plug-in .jar files.
Automation of the above process using Ant scripts, predefined Eclipse tasks or some such is left as an exercise for the reader. Also note another post with hints on how to debug helper applications started by the Eclipse launcher.



Debugging Eclipse helper applications
2009-05-07T14:34:00.001+02:00
Eclipse includes some command-line tools for platform developers, which can be launched by providing the -application option to the Equinox launcher JAR (formerly startup.jar). A need may arise to debug one of those helper applications. Proceed as follows:

Import the plug-in org.eclipse.equinox.launcher into workspace (with source code).
Import the plug-in containing the helper application into workspace (with source code). For example: org.eclipse.update.core contains the siteOptimizer application.
Create a debug configuration for the Java class org.eclipse.equinox.launcher.Main. Specify -Dosgi.install.area=/path/to/eclipse/installation/dir as a VM argument. Specify the remaining command-line arguments (e.g. -application ...).
Set a breakpoint at the entry point to the helper application (likely a method called run or start).
If the debugger asks for a source code location, point it to the workspace project.



Solution for missing startup.jar in Eclipse
2009-05-07T14:32:00.001+02:00
Some articles, for example Update Site Optimization, mention a file startup.jar used for launching Eclipse's helper command-line applications. However, this file no longer exists in newer versions of Eclipse (such as 3.4).
Solution: just replace references to the missing startup.jar with references to $ECLIPSE_HOME/plugins/org.eclipse.equinox.launcher_version.jar
(Insert the actual version found in your plugins directory).



Solution for no output from digestBuilder in Eclipse siteOptimizer
2009-05-07T14:08:00.005+02:00
The article Update Site Optimization provides instructions for creating optimized update sites in Eclipse to speed up downloads. While following those instructions, you may run into a problem that the digestBuilder component seems to run (displaying lines such as "Extracting locales from feature jar" and "Processing... feature jar"), but no output is ever written to the specified digestOutputDir. Alternatively, digest.zip might be created,
but missing some features.
Solution: make sure that each of your feature JAR files contains a file named feature.properties (even an empty one). As of Eclipse 3.4, features without this file are skipped altogether by the digestBuilder.
Further reading: Debugging Eclipse helper applications and the tutorial on how to optimize an Eclipse update site for P2.



Viewing internal representation of strings in Perl
2009-04-23T21:34:00.010+02:00
This article explains how to debug encoding problems arising from a wrong interpretation of the byte sequence stored as the internal representation of a string variable in Perl 5.8.x. Such problems can manifest themselves as invalid characters being printed or stored to a database under a variety of circumstances. The official Perl utf8 documentation explains how Perl handles string values behind the scenes, and experts advise to disregard the internal representation and just trust Perl to do the right thing (see also these recommendations on dealing with Unicode in Perl). While founded on good premises of early prevention, the ideal approach falls short when you have to diagnose real-life problems.
Here is what you need to know:
The value of every string variable is internally stored by Perl in a data structure with two fields: a sequence of bytes (called the "internal representation"), and a flag (called the "utf8 flag") indicating how the internal representation should be interpreted by functions that deal with characters (such as print, substr, and length).
If the flag is set, the internal representation is interpreted as a UTF-8 encoded string. If the flag is not set, the same byte sequence is interpreted as an ISO-8859-1 encoded (Latin1) string.
Problems arise when the flag for some reason does not match the actual encoding used in the internal representation.
The official documentation explains how to determine the flag's current value: utf8::is_utf8($string) or, if you want to remain compatible with Perl 5.8.0, Encode::is_utf8($string). But it does not reveal how to view the internal representation. There are several ways that do not work, such as Data::Dumper and a straightforward unpack('C*', $string). The solution is a combination of use bytes and unpack, as illustrated by the following sample code:
use Encode;

my $hash = { key1 => 'ä', key2 => 'ä' };

# Convert the internal representation from latin1 to utf8.
# This also turns on the utf8 flag on the value:
utf8::upgrade($hash->{key2});

print "str1: " . hexdump($hash->{key1}) . "\n";
print "str2: " . hexdump($hash->{key2}) . "\n";

if ($str1 eq $str2)
{
    print "str1 eq str2\n";
}

# Note that the output of Data::Dumper is not helpful:
use Data::Dumper;
print Dumper($hash);

# For a given string parameter, returns a string which shows
# whether the utf8 flag is enabled and a byte-by-byte view
# of the internal representation.
#
sub hexdump
{
    my $str = shift;
    my $flag = Encode::is_utf8($str) ? 1 : 0;
    use bytes; # this tells unpack to deal with raw bytes
    my @internal_rep_bytes = unpack('C*', $str);
    return
        $flag
        . '('
        . join(' ', map { sprintf("%02x", $_) } @internal_rep_bytes)
        . ')';
}
The correct encoding for the Unicode character No. 0x00E4 "latin small letter a with diaeresis", also known as "a umlaut" (ä) happens to be 0xE4 in ISO-8859-1 and 0xC3 0xA4 in UTF-8. Note that Data::Dumper displays the Unicode character number "\x{e4}" (which coincides with ISO-8859-1) rather than the internal byte representation. The program's output is shown below:
str1: 0(e4)
str2: 1(c3 a4)
str1 eq str2
$VAR1 = {
          'key1' => 'ä'
          'key2' => "\x{e4}",
        };
Once you determine that the utf8 flag is set incorrectly (mismatching the internal representation) your next questions are surely going to be:

Why is the flag set the wrong way?
How do I fix the flag - without also modifying the bytes of the internal representation?
You should first attempt to answer the first question, by examining your code (and very likely also the code of used modules). The second question essentially asks for "quick and dirty" workarounds, given that the various functions provided by the utf8 and Encode modules do alter the internal representation as their intended effect. Nothing is impossible in Perl, however:

Unset the utf8 flag
{ use bytes; $str = pack('C*', unpack('C*', $str); }
Set the utf8 flag
$str = pack("U0C*", unpack("C*", $str));

Checking that the internal representation and the value of the utf8 flag are synchronized is the basic first step in troubleshooting encoding problems in Perl. It ensures the consistency of data manipulated in memory. New pitfalls lie where the data has to leave the process, being subject to character encoding conversions in input/output operations (e.g. communicating with a database). However, that's another can of worms...



Debugging mod_perl with EPIC
2009-04-15T23:24:00.005+02:00
This article describes how to debug mod_perl applications using EPIC and explains some of the current shortcomings. The functionality was tested with EPIC version 0.6.33, mod_perl version 2.0.4, and Apache::DB version 0.14. It is important to note that while the EPIC debugger frontend is fairly stable and debugging mod_perl applications in principle is not very different from debugging remote Perl scripts, at the time of writing there is little experience with EPIC and mod_perl in particular. Furthermore, the procedures described here will appear a little kludgy, as they overload the already existing remote debugging functionality to serve mod_perl debugging. You should expect to invest some time into troubleshooting before you arrive at a working configuration.
Set up Apache::DB without EPIC
The first step is to make sure that the interactive command-line debugger works as expected (without EPIC). You can find a more detailed description of Apache::DB in CPAN and in mod_perl documentation. In the following, I assume that you have an example Apache site configured as follows:
Alias /modperl/ /home/jpl/modperl/
    
PerlModule ModPerl::Registry
<IfDefine PERLDB>
    PerlRequire /home/jpl/modperl/db.pl
</IfDefine>
PerlPostConfigRequire /home/jpl/modperl/startup.pl

<Location /modperl/>
    SetHandler perl-script
    PerlHandler ModPerl::Registry
    <IfDefine PERLDB>
        PerlFixupHandler Apache::DB
    </IfDefine>
    Options +ExecCGI
    PerlSendHeader On
    allow from 127.0.0.1
    deny from all
</Location>
The file /home/jpl/modperl/db.pl referenced in the above configuration contains:
use APR::Pool (); 
use Apache::DB (); 
Apache::DB->init();
The file /home/jpl/modperl/startup.pl contains whatever is needed to initialize the environment for your mod_perl scripts. For example, we should define @INC there:
use lib qw(/home/jpl/modperl/lib);
1;
Apache::DB requires that Apache is started with the -X (single-process) option. I use apache2ctl -k start -X -DPERLDB, which leaves the Apache process running in the foreground and enables the debugging-related directives by defining the PERLDB token. Whenever a script /modperl/*.pl is loaded in the browser, the terminal running apache2ctl gets an interactive debugger prompt. So far, this is all more or less standard mod_perl configuration, no EPIC involved.
                                                                                                                                                                      Create a "Perl Remote" debug configuration in EPIC
                                                                                                                                            In Eclipse, open the dialog Run/Debug Configurations... to create a launch configuration of type "Perl Remote" with the following settings:
                                                     
Project The Perl project containing your scripts and modules.
File to execute It doesn't matter - choose any script from the project.
                                                                                                      Local Host IP It doesn't matter - enter 127.0.0.1.
Target Host Project Installation Path Enter the file system path on the remote machine under which your Perl project is deployed (eg. /home/jpl/modperl). If you are running Apache on the same machine as Eclipse, enter the path displayed in Project Properties as "Location". If you notice that breakpoints are ignored, this is a key setting to tweak, more on that later. 
Port This is the port on which EPIC is going to listen for a connection from the debugger running on the target Apache host. Enter a port number that is not firewalled (in case of doubt, test connectivity with netcat or the like!). Default is 5000.
Capture Output It has to be unchecked, as you don't want EPIC to attempt hijacking of STDOUT and STDERR of your scripts!
Create Debug Package It should be checked on the first invocation (see below). It may be unchecked later on.
Debug Package File Path Enter the path to a new file that EPIC will create locally when the debugging session is started, for example: /tmp/epicdb.zip. This ZIP file will contain a custom version of perl5db.pl, which must be used instead of the Apache/perl5db.pl in order to support setting breakpoints (it refers to EPIC's epic_breakpoints.pm module, also included in the ZIP file). More on that in the next section.

Run the debug configuration. You should see a Remote Perl Script item in the Debug view (switch to the Debug perspective if necessary). Also, on disk, the ZIP archive mentioned above should now be available. Terminate the debugger session now, as we are not done with adjusting the configuration yet.
Install the EPIC helper modules on the target host
Copy the ZIP archive created in the previous step to the Apache host, create a new directory there and unpack the ZIP archive into it. You will notice that all project files are contained in it. However, you should only leave
*epic*.pm and perl5db.pl in the target directory - remove all other files and subdirectories.
Redirect Apache::DB to contact EPIC
Edit your Apache configuration like so:
Alias /modperl/ /home/jpl/modperl/

PerlModule ModPerl::Registry
<IfDefine PERLDB>
    PerlSetEnv PERLDB_OPTS "RemotePort=127.0.0.1:5000 DumpReused ReadLine=0 PrintRet=0"
    PerlSetEnv PERL5DB "BEGIN { $DB::CreateTTY=0; require '/path/to/EPIC/provided/perl5db.pl'; }"
    PerlRequire /home/jpl/modperl/db.pl
</IfDefine>
PerlPostConfigRequire /home/jpl/modperl/startup.pl

<Location /modperl/>
    SetHandler perl-script
    PerlHandler ModPerl::Registry
    <IfDefine PERLDB>
        PerlFixupHandler Apache::DB
    </IfDefine>
    Options +ExecCGI
    PerlSendHeader On
    allow from 127.0.0.1
    deny from all
</Location>

/path/to/EPIC/provided/perl5db.pl should be adjusted to point to where you unpacked the ZIP archive.
RemotePort should be adjusted to include the IP address and port where Eclipse/EPIC are listening for debugger connections.
Edit db.pl to include EPIC helper modules
Add the directory in which perl5db.pl and epic_breakpoints.pm are located to @INC. I suggest that you edit the file db.pl, which was mentioned above:
use lib qw(/path/to/EPIC/provided);
use APR::Pool ();
use Apache::DB ();
Apache::DB->init();

Test debugging without breakpoints
In EPIC Preferences, enable the "suspend at first line" option. Launch the Remote Perl debug configuration created earlier. On the target host, restart Apache in single-process mode. Load one of your scripts in the browser. You should see in EPIC that the debugger suspends in one of the handler modules (eg. Apache::Registry). From here on, you could step through the code to eventually enter your scripts and modules. Resume execution, try reloading the script or loading another script. EPIC should suspend again. If this doesn't work, there is no sense of going further. For troubleshooting, you can enable Perl debugger console in EPIC Preferences.
The proper way to terminate the debugging session is stop the Apache process. EPIC should detect it and terminate the session automatically. If you attempt terminating the session from EPIC, you may get into trouble and may have to kill the Apache process with kill -9 later. In case of problems, make sure that you have no runaway Apache processes. The relative order of starting the EPIC debugging session and the Apache process doesn't matter. Obviously, before you run a script to be debugged by making a request in the browser, both the EPIC debugging session and the Apache process must be running.
Test breakpoints in your own modules
Do not set breakpoints in scripts just yet. Set a breakpoint in one of your own modules and check whether EPIC properly suspends on it. If the breakpoint is ignored, check the file workspace/metadata/.log for warnings like "Could not map local path ... to a remote path". If they appear for the file in which you set breakpoints, you should try adjusting "Target Host Project Installation Path" in the Perl Remote launch configuration. Watch out for possible path variations due to symlinks. Essentially, the path entered in the launch configuration should correspond to the path used by Perl internally to refer to files on the remote host. You can see those internal paths if you enable debugger console in EPIC preferences and step through your code. For example, I had trouble with breakpoints being ignored due to the Perl debugger referring to files as /home/jpl/modperl/hello.pl, while the location displayed in file properties in EPIC was /mnt/data/home/jpl/modperl/hello.pl. The project's include path configured in EPIC and the actual include path on the remote host are also used in mappings. As a general rule, the less the remote host differs from the development machine on which EPIC is running, the better.
Test breakpoints in scripts
Breakpoints in scripts don't work out-of-the-box with mod_perl and EPIC. The reason is that EPIC gets no opportunity to actually set breakpoints in scripts, which are loaded by mod_perl in a custom way and converted dynamically into packages. Fortunately, there is a little trick which allows suspending in scripts. Edit the target script to include $DB::single = 1; immediately before the line on which you want to suspend. Also don't forget to set an EPIC breakpoint as usual, like so:
$DB::single = 1;
print "suspend before print\n"; # set EPIC breakpoint on this line!
The debugger should now suspend on the correct line.



Workaround for Mailman list footer sent as attachment
2009-03-15T15:04:00.001+01:00
Summary: The popular open source mailing list software GNU Mailman does not play well with the popular mail client Thunderbird running under the German version of the popular operating system Windows. When you send a message to a Mailman-managed list, the list footer may be included as an attachment ("MIME part") rather than appended to the message. Mailman developers deny responsibility for (solving) the problem.
The short story is that even if your mail client sends a plain text message, but it happens to be in a different character encoding than the ISO-8859-1 encoding expected by Mailman, the footer will be attached improperly. For German Thunderbird/Windows, you can change the default encoding for outgoing messages from ISO-8859-15 (German with Euro) to ISO-8859-1 (German) to work around this problem. The option that you should change is "mailnews.send_default_charset"
in Advanced preferences (also accessible through about:config).
The problem is not solved once and for all by the above workaround, as it depends on the configuration of each sender's mail client (but not each recipient's mail client). However, the workaround is sufficient if you use the mailing list for one-to-many communication.
I'll spend the rest of this article dissecting the problem on a more general level, as it seems like a nice example for discussing some implicit differences between the worlds of professional and amateur software development (aka work vs. play). 
The problem is a special case of an old and well-understood issue, elaborated at length in the FAQ by the Mailman developers. Unfortunately, the four solutions proposed there are all unsatisfactory (my comments in italic):

Configure Mailman to remove the msg_header/msg_footer. But I want this footer to appear! Essential feature!
Configure Mailman to strip all HTML and MIME formatting and send out all messages as text-only (text/plain). I've already done that, and it didn't help.
Get everyone to change the MUA they use to one that is more HTML/MIME-aware. Yeah, dream on.
Live with the problem. Come on guys, is attaching a few lines of ASCII text to a text-only email really so difficult?

The answer to the last question might be: "Yes, if you want to make it work universally". But users don't care about what works universally, they care what works for them, out of the box. The developers' standpoint is an example of a more general behavioral pattern common among open source programmers (as I know well from my own experience as the volunteer maintainer of EPIC). What we, the developers, are saying is, in order
of increasing detail:

Don't bother me.
Works as designed!
You have no idea how complicated this code is and how much time we spent on it already.
Solving this "little" problem of yours would break n different exotic use cases of which you're completely unaware.

Of course it would be more up-front to say:

Oops, we screwed up. But we are lazy/not in mood to fix it for you. If you don't like it, find a better product/service. Sorry.

...which is quite a legitimate reply for a development-is-my-hobby-and-you-are-not-my-customer type of project.
In a professional setting, the developers answer to the paying client. Either a fix or a satisfactory workaround must be delivered on time, no matter whether it seems convenient and aesthetically pleasing to the developers and whether the blame for the problem lies in our code or elsewhere. We are hired to shield our clients from this kind of deliberation and the only factor that we might call to our rescue is whether the client's request seems to be a valid business requirement without conflict with her other priorities. Being able to send mails using Thunderbird and see footers properly attached by Malman certainly meets these criteria.



Debugging serialization in SOAP::Lite
2009-03-05T15:07:00.002+01:00
Scenario: you are using the Perl SOAP::Lite module as a SOAP client, and want to invoke a web service operation which accepts a complex data type. You provide SOAP::Lite with an appropriately structured SOAP::Data object that represents the value of your type. However, on the server side you notice that xsi:nil is transmitted instead of the value.
The solution is to define a seralizer method for your complex type, like so:
sub SOAP::Serializer::as_name_of_your_complex_type
{
    my $self = shift;
    my $value = shift;
    my $name = shift;
    my $type = shift;
    my $attr = shift;

    return [$name, { 'xsi:type' => 'tns:name_of_your_complex_type', %$attr }, $value];
}
The rest of this post elaborates how one would arrive at this solution and provides a short study of SOAP::Lite internals.
The perils of understanding SOAP::Lite
Debugging SOAP::Lite is not easy. In fact, I bet that the author of SOAP::Lite does not use a debugger himself. Apparently, he didn't spend a lot of time pondering the cruelty to his readers inflicted by a whole arsenal of Perl tricks. Let me enumerate several issues:

The code is long. This reduces your chances of finding a portion relevant to your problem by the sheer luck of skimming over it.
It heavily relies on Perl automatic (predefined) variables. Your memory will be severely strained if you try to read this code. The author possibly assumes that Perl is your only language and mother tongue, too.
It uses generated code and late initialization to great extent. Of course, you can't set breakpoints in the generated code. It's also rather difficult to SEE the generated code in all its glory, for this requires you to first establish the numerous points of its generation. You will also get dizzy from the frequent back-and-forth transfers of control between subroutines that implement actual functionality and generic "internal plumbing" subroutines that have little to do with the main thread of execution. Forget stepping through SOAP::Lite code. At best, you can stumble through it.
It uses nested expressions a lot - long live functional programming! Too bad that it defeats the idea of setting line breakpoints.
It follows the "self-documenting genius code" style. Natural language explanations of the code are unnecessary. Even worse, it doesn't contain any specifications (parameter/return value/contracts) for the subroutines. You have to infer each subroutine's intent and valid invocation scenarios solely based on its implementation. Surely, there is a method to this madness, but it's up to you to find it.

All that said, one must admit that SOAP::Lite has been around for a very long time, works very well (until it ceases working, that is), it is the
only such comprehensive SOAP module around, and it is of course free. These factors mean that we have to adjust to its author's preferred programming style rather than vice versa (and also thank him for sparing us the effort of implementing the SOAP protocol ourselves).
A walkthrough toward generated code
There are plenty of ways to interact with SOAP::Lite. For the following descriptions, assume that you use something like that:
my $service = SOAP::Lite        
        ->service("$url/wsdl/inventory.wsdl")
        ->proxy("$url/cgi-bin/soap_server.pl");

my $result = $service->dispatch_to(
        $test_user,
        $test_password,
        'inventory_update',
        $args
        );

...where $test_user, $test_password are strings and $args is a SOAP::Data object containing the value of a complex type.
You begin your interaction with SOAP::Lite by invoking the static service method. This method is a factory method - it invokes the SOAP::Lite constructor to create a new SOAP::Lite object. You don't get a SOAP::Lite instance from it, though. Rather, the object is blessed into a (generated) subclass whose name is based on the name attribute of the wsdl:service element in the WSDL you specified as parameter. In our example case, the subclass is called 'dispatch_to_service'.
Two noteworthy attributes of the newly created object are _serializer and _deserializer, of type SOAP::Serializer and SOAP::Deserializer, respectively. Other attributes' names also start with an underscore.
The created service object at first appears to have no methods of its own. Various methods are added to this object at various points of execution, though. For example, when you attempt to invoke the method proxy, the code generator kicks in. In this case, a subroutine (?) named SOAP::Lite::BEGIN generates non-underscore-named accessor (get/set) methods for all (most of?) the underscore-named attributes. Also generated is the 'proxy' method, which delegates to the the same-named method on the SOAP::Transport object stored in the _transport attribute. This one in turn creates a SOAP::Transport::HTTP::Client object, which is stored in the _proxy attribute of the service object and henceforth returned whenever you call $service->proxy.
Within your invocation of SOAP::Lite::service, the generation of stubs for wsdl:operations of the service (that is, the actual business logic methods) also occurs. More precisely, the following line from Lite.pm is responsible for invoking the generator:
my %services = %{$self->schema->parse(@_)->load->services};

The subroutine SOAP::Schema::load iterates over all the operation names and invokes generate_stub for each operation. The generated stub code is stored in the _stub attribute of a SOAP::Schema object. Actually, the operations are not even placed as normal methods in the generated stub code, but rather (again) generated using the AUTOLOAD mechanism by the stub code. Here is an example of what it looks like in all its generated glory. Actually getting to the point of seeing this code took me a few hours debugging:
package dispatch_to_service;
# Generated by SOAP::Lite (v0.710.08) for Perl -- soaplite.com
# Copyright (C) 2000-2006 Paul Kulchenko, Byrne Reese
# -- generated at [Thu Mar  5 14:05:39 2009]
# -- generated from http://localhost/wsdl/inventory.wsdl
my %methods = (
dispatch_to => {
    endpoint => 'http://localhost/cgi-bin/soap_server.pl',
    soapaction => 'http://www.plosquare.com/wsdl/WebServices#dispatch_to',
    namespace => 'http://www.plosquare.com/wsdl/WebServices',
    parameters => [
      SOAP::Data->new(name => 'user', type => 'xsd:string', attr => {}),
      SOAP::Data->new(name => 'password', type => 'xsd:string', attr => {}),
      SOAP::Data->new(name => 'function', type => 'xsd:string', attr => {}),
      SOAP::Data->new(name => 'args', type => 'tns:inventory_argument_type', attr => {}),
    ], # end parameters
  }, # end dispatch_to
); # end my %methods

use SOAP::Lite;
use Exporter;
use Carp ();

use vars qw(@ISA $AUTOLOAD @EXPORT_OK %EXPORT_TAGS);
@ISA = qw(Exporter SOAP::Lite);
@EXPORT_OK = (keys %methods);
%EXPORT_TAGS = ('all' => [@EXPORT_OK]);

sub _call {
    my ($self, $method) = (shift, shift);
    my $name = UNIVERSAL::isa($method => 'SOAP::Data') ? $method->name : $method;
    my %method = %{$methods{$name}};
    $self->proxy($method{endpoint} || Carp::croak "No server address (proxy) specified")
        unless $self->proxy;
    my @templates = @{$method{parameters}};
    my @parameters = ();
    foreach my $param (@_) {
        if (@templates) {
            my $template = shift @templates;
            my ($prefix,$typename) = SOAP::Utils::splitqname($template->type);
            my $method = 'as_'.$typename;
            # TODO - if can('as_'.$typename) {...}
            my $result = $self->serializer->$method($param, $template->name, $template->type, $template->attr);
            push(@parameters, $template->value($result->[2]));
        }
        else {
            push(@parameters, $param);
        }
    }
    $self->endpoint($method{endpoint})
       ->ns($method{namespace})
       ->on_action(sub{qq!"$method{soapaction}"!});
  $self->serializer->register_ns("http://schemas.xmlsoap.org/wsdl/","wsdl");
  $self->serializer->register_ns("http://www.w3.org/2001/XMLSchema-instance","xsi");
  $self->serializer->register_ns("http://schemas.xmlsoap.org/soap/encoding/","soapenc");
  $self->serializer->register_ns("http://www.w3.org/2001/XMLSchema","xsd");
  $self->serializer->register_ns("http://www.plosquare.com/wsdl/WebServices","tns");
  $self->serializer->register_ns("http://schemas.xmlsoap.org/wsdl/soap/","soapbind");
    my $som = $self->SUPER::call($method => @parameters);
    if ($self->want_som) {
        return $som;
    }
    UNIVERSAL::isa($som => 'SOAP::SOM') ? wantarray ? $som->paramsall : $som->result : $som;
}

sub BEGIN {
    no strict 'refs';
    for my $method (qw(want_som)) {
        my $field = '_' . $method;
        *$method = sub {
            my $self = shift->new;
            @_ ? ($self->{$field} = shift, return $self) : return $self->{$field};
        }
    }
}
no strict 'refs';
for my $method (@EXPORT_OK) {
    my %method = %{$methods{$method}};
    *$method = sub {
        my $self = UNIVERSAL::isa($_[0] => __PACKAGE__)
            ? ref $_[0]
                ? shift # OBJECT
                # CLASS, either get self or create new and assign to self
                : (shift->self || __PACKAGE__->self(__PACKAGE__->new))
            # function call, either get self or create new and assign to self
            : (__PACKAGE__->self || __PACKAGE__->self(__PACKAGE__->new));
        $self->_call($method, @_);
    }
}

sub AUTOLOAD {
    my $method = substr($AUTOLOAD, rindex($AUTOLOAD, '::') + 2);
    return if $method eq 'DESTROY' || $method eq 'want_som';
    die "Unrecognized method '$method'. List of available method(s): @EXPORT_OK\n";
}

1;

Accordingly, if you want to debug your stubs, you should incorporate your debugging output in the generator routine SOAP::Schema::generate_stub. Back in the beginning of the post, the problem was that a complex data structure was being serialized into xsi:nil rather than into the provided value. Inserting debugging output would have shown that the line
my $result = $self->serializer->$method($param, $template->name, $template->type, $template->attr);

in the generated stub doesn't yield the expected value for the 'args' parameter and would provide a clue to the custom serialization method as a method of solving the problem.
Hopefully you enjoyed this little trip into SOAP::Lite internals as much as I did making it on my own (that is to say, just a bit ;-). Seriously, this post serves to capture several facts about SOAP::Lite that might aid debugging if (when) the next such necessity arrives.



Exporting data from Microsoft Access to MySQL using a linked table, trigger, UDF
2009-03-04T10:32:00.003+01:00
This article describes a simple, but effective integration solution for transferring data on user request from an MS Access database to a MySQL database hosted on a remote web server. The data transfer occurs over an encrypted connection through the Internet, which creates some performance challenges. On the receiving side, the data has to be post-processed, which is achieved using a MySQL trigger and a user-defined function to notify a background daemon process.
The solution is outlined below:

First off, ensure ODBC connectivity between the MySQL Access database and the MySQL database on the remote server. Install MyEnTunnel on the Windows machine to tunnel port 3306 from the local machine to the remote server over an encrypted (ssh) link. Configure MyEnTunnel to keep the connection open at all times. Set up an ODBC data source on the Windows machine to connect to the appropriate database at localhost:3306 (now tunneled to the remote server), using appropriate credentials. Note: I had some trouble configuring MyEnTunnel as a Windows service on Vista (though it worked ok on Windows XP). Adding it as a normal autostart program was a viable workaround.
Based on the connection from the previous step, you can already create a linked table in Access which is mapped to a pre-existing MySQL table. Inserting rows into that table will insert them into MySQL. However, if you try inserting >1000 rows through a DSL connection, you will notice that the whole setup is unbearably slow (several minutes). Same applies to deleting rows. Some suggest setting up pass-through queries to work around the problem. I have not seriously tried this. Instead, I chose to reduce the number of INSERTs necessary by using a single linked table which acts as a "mailbox" on the MySQL side for receiving a "message" (long string) from MS Access. This message is deciphered by a remote daemon process.
A naive approach would be to have the mailbox table contain a single column of type "text" and accept a single row. This won't work, MS Access will truncate everything below 64 KB or even below. You have to split the message into chunks. I chose to have 32 KB chunks. The layout of the mailbox table is as follows:CREATE TABLE `mailbox` (
  `id` int(4) NOT NULL auto_increment,
  `msg` text,
  `_ts` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
  `last_row` int(1) default '0',
  PRIMARY KEY(`id`)
)

To transmit the chunked message, first a DELETE FROM mailbox is executed, then a series of rows, each containing up to 32 KB msg payload are inserted, and all but the last row have last_row=0. The chunking is done by a VBA macro, which basically executes a query in Access, concatenates the rows (and columns) into a string, finally divides up the string into appropriately sized chunks.  

The following steps will make the server do something in response to the received data. Generally, you can react to database changes using triggers (provided that your MySQL version is recent enough). What we'd like to do is to have a trigger wake up a daemon process, which can then read the mailbox table, stitch together the received message chunks, and process the message. For that, we'll to add three more pieces to the puzzle: a dummy table, a trigger, and a user-defined function (UDF).
The dummy table looks as follows:CREATE TABLE `dbg_tmp_dummy` (
  `dummy` varchar(256) default NULL
)

It should contain a single row, value doesn't matter. It will become apparent next why the dummy table is required.

In order to notify an external process, I chose to use the tried and true UNIX named pipe mechanism (signals could be used just as well, but you'd then have to run the daemon process under MySQL identity). MySQL is supposed to write a character to a FIFO file, which the daemon process is reading from. MySQL doesn't support writing to files out-of-the-box without overwriting the file (the built-in export function will complain if the target file already exists). To work around this, a little MySQL user-defined function has to be implemented in C, so that the following incantation is possible: SELECT writefile('/tmp/fifo.pipe', 'x'); with the effect of a single byte 'x' being appended to /tmp/fifo.pipe. Given that we are dealing with a named pipe, and that the daemon process might not be listening, the write has to be a non-blocking one. I will skip the implementation details here. Rest assured that the UDF is a simple one. If you are new to UDFs, read the MySQL manual and have a look at udf_example.c shipped with the MySQL source code. Essentially, you compile the C file into a .so library, place the library in a directory searched by MySQL, use CREATE FUNCTION to register the function in your database.

Finally comes the trigger part. To define the trigger, connect to the target database using the command-line MySQL client and enter the following:delimiter |
CREATE TRIGGER mailbox_trigger AFTER INSERT ON mailbox
FOR EACH ROW BEGIN
UPDATE mailbox SET dummy=writefile('/tmp/fifo.pipe', 'x') WHERE NEW.last_row=1;
END;
|

The above code is quite self-explanatory. The important things to note is that you have to use the special delimiter, you can't just enter the trigger definition having the standard semicolon as the delimiter (you'd get a syntax error). It is now apparent why the dummy table is needed. It is impossible to execute a SELECT statement within a trigger which is not tied to an INSERT/UPDATE. Finally, note how the last_row attribute is used to only send the notification after the full message has been received.

The daemon process might be a Perl script which reads from the pipe:sub wait_for_notification
{
    if (! -p $pipe_path) # create pipe if necessary
    {
        unlink($pipe_path);
        my $prev_umask = umask(077); # rw for owner (mysql)
        system("mknod $pipe_path p", 1, 1) || die "mknod: $!";
        umask($prev_umask);
    }
    open(FL, "<$pipe_path") || die "open: $!";
    for (;;)
    {
        my $buf = '';
        my $ret = sysread(FL, $buf, 1);
    
        if (!defined($ret))
        {
            die "sysread $pipe_path: $!";
        }
        elsif (!$ret) # EOF: we read something
        {
            close(FL) || die "close $pipe_path: $!";
            return;
        }
    }
}

Beware that notifications will be lost if MySQL writes to the pipe while the daemon is not reading it. For my particular use case, this is not an issue, as the daemon can also periodically poll the database and the notification volume is very low. You could modify the solution to use shared memory with appropriate locking instead of pipes to make sure that no race conditions are possible.   




Solution for Eclipse CVS merge conflicts
2009-02-16T22:34:00.003+01:00
Problem: Eclipse's CVS merge feature (despite all your efforts) keeps showing tons of pseudo-conflicts (red conflict symbol with a superimposed white plus or cross sign, whatever it might mean). Solution: don't use Eclipse (for merging). Instead, fall back on the good old command-line CVS client; it's not that difficult.
Change to your project directory and issue a command like:
cvs -q update -d \
    -j "VERSION_2_2:2008-11-24" \
    -j VERSION_2_2 &> /tmp/merge.log

Here VERSION_2_2 is your source branch, from which the merging
should occur. The first -j also contains a start tag (or date).
Inspect merge.log for mentions of "warning" and "conflict". Inspect the reported files and merge manually around the standard CVS conflict markers, i.e. >>>>, ==== and <<<<). Finally, delete all files whose names match the pattern .#* (created by the CVS client to indicate which files were affected by merge).
There is one catch here - the extssh access mode routinely used by Eclipse is not recognized by the command-line CVS client, so you will have to replace :extssh:  strings by :ext: in all CVS/Root files within your project and set the environment variable CVS_RSH=/usr/bin/ssh to emulate Eclipse's behavior (after you're
done revert the replacements).



Subprocess post-installation script returned error exit status 1
2009-01-22T11:17:00.002+01:00
Scenario: while installing a package in Debian, we get into the following situation:
debian:~# apt-get install bugzilla
Reading Package Lists... Done
Building Dependency Tree... Done
bugzilla is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 1272 not upgraded.
1 not fully installed or removed.
Need to get 0B of archives.
After unpacking 0B of additional disk space will be used.
Setting up bugzilla (2.22.1-2) ...
dbconfig-common: writing config to /etc/dbconfig-common/bugzilla.conf
dpkg: error processing bugzilla (--configure):
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 bugzilla
E: Sub-process /usr/bin/dpkg returned an error code (1)

This is yet another "observability" problem, as clearly too little information is provided to determine the problem's cause.
The correct way to proceed is to trace the execution of the failed post-installation script, but first we need to find out where it is and how it is called. Note that the apt-get install internally runs dpkg --install and, as we can learn from dpkg's man page, dpkg --configure. The post-installation script is invoked by dpkg --configure. To see how it is done, we use strace:
debian:~# ( strace -f dpkg --configure bugzilla 2>&1 ) \
| grep execve | grep post
[pid  6289] execve("/var/lib/dpkg/info/bugzilla.postinst",
["/var/lib/dpkg/info/bugzilla.post"..., "configure", "2.20.1-1"],
[/* 47 vars */] 
[pid  6289] execve("/usr/share/debconf/frontend",
["/usr/share/debconf/frontend", "/var/lib/dpkg/info/bugzilla.post"...,
"configure", "2.20.1-1"], [/* 45 vars */]) = 0
[pid  6310] execve("/var/lib/dpkg/info/bugzilla.postinst",
["/var/lib/dpkg/info/bugzilla.post"..., "configure", "2.20.1-1"],
[/* 46 vars */]) = 0

Based on the first logged execve, we try the following command:
/var/lib/dpkg/info/bugzilla.postinst configure 2.20.1-1; echo $?
and we see that it in fact reproduces the error reported by dpkg --configure. From here we can rely on generic shell script tracing or debugging techniques to determine the cause.



Renaming rausb0 to wlan0
2009-01-18T12:41:00.006+01:00

Problem: When a USB wireless device using the rt73 driver is plugged in under Linux, it appears (e.g. in ifconfig) as rausb0 rather than the more familiar device name wlan0.
Solution: Add the following line to your /etc/udev/rules.d/70-persistent-net.rules in order to rename the device:ACTION=="add", KERNEL=="rausb0", NAME="wlan0"


To diagnose this sort of problems, launch udevadm monitor before running modprobe rt73. The correct output of udevadm monitor while the renaming rule is in effect should look as follows:
UEVENT[1232279287.492254] add      
/module/rt73 (module)
UEVENT[1232279287.494837] add      
/bus/usb/drivers/rt73 (drivers)
UEVENT[1232279287.496842] add      
/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1:1.0/rausb0 (net)
UEVENT[1232279287.501910] add      
/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1 (firmware)
UDEV  [1232279287.505345] add      
/module/rt73 (module)
UDEV  [1232279287.508952] add      
/bus/usb/drivers/rt73 (drivers)
UDEV  [1232279287.534015] add      
/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1:1.0/wlan0 (net)
UEVENT[1232279287.555346] remove   
/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1 (firmware)
UDEV  [1232279287.558466] remove   
/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1 (firmware)
UDEV  [1232279287.561163] add      
/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1 (firmware)



ModeLine ignored by xorg display driver
2008-12-30T13:44:00.005+01:00
In the recent past, I used to switch screen resolutions in XFree/X.org by pressing the CTRL-ALT-minus or CTRL-ALT-plus keys. For each supported mode, there used to be a ModeLine entry in xorg.conf and those entries were referenced by the "Screen" section. As often happens when technology "advances", on my new Dell D830 notebook (using xorg 1:7.3+18 from Debian 'testing', GM965/GM960 graphics controller, 'intel' display driver), this tried and true method no longer works. The magic key combo is ignored altogether, as is adding ModeLines to xorg.conf. No warnings in /var/log/Xorg.0.log nor anywhere else are recorded, AFAICS. The standard xorg.conf seems almost empty, which in itself is a good thing (less configuration means less opportunity for mistakes). The bad thing, obviously, is that invalid or deprecated (?) configuration entries are silently skipped without a clue for the user about what is going on inside the black box, whether its behavior is normal or not, and which alternatives might be available. This is an example of what I like to call an "observability" problem. Sometimes a careless implementation is to blame, but such problems may also have their roots in software designers' inability to correctly foresee and specify the possible variance in implementations.
Fortunately, there is an easy solution to ignored ModeLines. xorg includes a program called xrandr, which can be used to test and programmatically set a resolution, as described in the Ubuntu Wiki article X/Config/Resolution. This seems to be a substitute for the traditional ModeLines in xorg.conf. Although the Wiki article mentions that xorg.conf can still be used for specifying resolutions, this is simply not the case in my setup.



Multi-boot configuration for Vista, Windows XP, Linux
2008-12-30T00:11:00.010+01:00
I bought my (Dell) notebook with Windows Vista preinstalled. Because some of my customers stay loyal to Windows XP and I prefer Linux for flexibility, I wanted to also install both of these OSes on the same disk, so as to have a choice of each operating system through the GRUB boot menu (Linux being the dominant OS and the other two secondary choices). This article contains a record of my installation with some warnings and explanations mixed in, unlike those "easy" HOWTOs that omit the gory details and leave you wanting if anything goes wrong in the process or your preconditions differ slightly.
First of all, it is important to note that the final configuration which I settled upon does not include all three OSes within a single GRUB boot menu. Instead, the menu contains two categories of items: Linux (one item per kernel configuration) and Windows (just one item for booting either Windows XP or Vista). The Windows menu item instructs GRUB to delegate control to Vista's own boot loader, which then offers a second choice between Vista and XP.
The entry for the Windows menu item in /boot/grub/menu.lst is trivial:
title           Windows Vista/XP
root            (hd0,2)
savedefault
makeactive
chainloader     +1

The partition table in the final configuration looks as follows - (hd0,2) in GRUB-speak refers to /dev/sda3:
  Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          23      184716   de  Dell Utility
/dev/sda2              24        1069     8401995    7  HPFS/NTFS
/dev/sda3   *        1070        7494    51602432    7  HPFS/NTFS
/dev/sda4            7495       29863   179678992+   5  Extended
/dev/sda5   *        7495        9925    19526976   83  Linux
/dev/sda6            9926       10411     3903763+  82  Linux swap / Solaris
/dev/sda7           10412       29863   156248158+  83  Linux


/dev/sda1 "Diagnostics" partition; this is a small FAT16 partition preinstalled by Dell. It contains programs for testing memory etc.
/dev/sda2 The Windows XP partition; originally the same block range  used to hold a ~10 GB "recovery" partition, which was apparently too small for Vista for its intended purpose.
/dev/sda3 The Windows Vista partition; originally it occupied the rest of disk and had to be shrunk to make room for Linux.
/dev/sda4 The extended partition which holds /dev/sda5, /dev/sda6, /dev/sda7. It is very important to create this partition because any disk can only hold 4 primary partitions. If you forget this little quirk, you will likely end up with large amounts of unpartitioned (that is, unusable) disk space.
/dev/sda5 The Linux boot/root partition.
/dev/sda6 The Linux swap partition.
/dev/sda7 The Linux "data" partition, which in my case holds an encrypted file system for /home and /usr/local (not quite relevant)

The non-trivial part of the setup concerns the order of installation and the steps required to make both boot managers work correctly. The same approach might be used to install in a different order. The key piece of information to keep in mind is which step overwrites which boot records.

Before installing anything, I made snapshots of the Master Boot Record (/dev/sda) as well as of each partition's initial sectors using a Knoppix CD. After booting from the CD, I issued commands like dd if=/dev/sda1 of=/path/to/sda1/backup/file bs=512 count=32 for each partition and the main MBR. To store the backup files, I used the /dev/sda1 "diagnostics" partition. A USB stick could be used just as well. I heuristically chose to back up the first 32 sectors of each partition, not being sure how many of them really need to be protected. This number has later turned out sufficient.
I began by resizing Vista's own partition using Vista's disk management tool. It didn't allow me to go below 50 GB. I used the same tool to delete the /dev/sda2 "recovery" partition in order to create a gap for installing Windows XP. (By the way, an attempt to use the Partition Magic tool to scale down Windows XP below 40 GB and move it to the beginning of disk resulted in an unbootable system; now I believe PM can't deal at all with Vista.)
I installed Windows XP, creating a new partition for it during the installation process. Actually, I had to first create a custom Windows XP installation CD based on the original one, in order to include some missing SATA disk drivers. The Windows XP installer did not just alter the MBR, but it also overwrote the beginning of the Vista's /dev/sda3 partition. Accordingly, I was unable to boot Vista after this step (instead Windows XP booted itself).
Using the Knoppix CD, I created another set of boot sector backup files (just in case; those were not used later on).
I proceeded to install Linux (Debian), taking care to use the "expert mode" to create an extended partition. (It later turned out that fdisk is too dumb to expand this partition in order to accommodate new logical partitions; I recommend cfdisk instead.)
At this point, I had a configuration which booted Linux, and (if I recall correctly) Windows XP, based on an entry in GRUB's menu.lst created by the installer - the same entry I pasted in the beginning of this post.
Having booted into Linux, I created another set of boot sector backup files (again, just in case).
The really interesting part begins now. To "revive" Vista's boot loader, I restored with dd the 16KB backup of Vista's partition I made in Step 1. This left me being able to boot Linux and Vista, but not Windows XP. (It is noteworthy that partition numbering changes when you delete a partition. For example, /dev/sda1, /dev/sda2, /dev/sda3 became /dev/sda1, /dev/sda2 after deleting the "recovery" /dev/sda2 partition. The original ordering was restored after installing Windows XP into /dev/sda2, though.)
In Vista, I installed the free EasyBCD tool, which lets you edit Vista's own boot loader entries. Here I added a "Windows XP" menu entry and pointed it to the partition with Windows XP (which is called D: from Vista's own perspective).
At this stage, my attempts to boot into Windows XP using Vista's boot loader failed with an error message about missing NTLDR. To fix this, following advice from the EasyBCD forum, I booted into Vista and copied c:\ntldr.dll to Windows XP's d:\ntldr.dll (where it was, in fact, missing). In order to see the file c:\ntldr.dll in Vista, I had to tweak some settings regarding displaying hidden and system files.

Having completed all of the above steps, I was able to boot all three OSes - Linux directly using GRUB, and the two versions of Windows by redirecting GRUB into Vista's boot loader. It might not be the nicest possible configuration (requires a few extra key strokes during the boot process), but it works.



Solution for Gtk-WARNING **: ... : wrong ELF class: ELFCLASS64
2008-12-29T21:34:00.007+01:00
(eclipse:13576): Gtk-WARNING **: /usr/lib/gtk-2.0/2.10.0/engines/libbluecurve.so: wrong ELF class: ELFCLASS64
If you get the above error message in a 64 bit Linux system while trying to run a 32 bit application using GTK 2 (like Eclipse), it means that GTK is picking up the wrong (64 bit) version of a plugin library. The first step is, of course, to check that the 32 bit version of the library is available (example location in Debian/Ubuntu; I first extracted the library from the i386 package and moved it to this location):
jpl@debian:~$ file /emul/ia32-linux/usr/lib/gtk-2.0/2.10.0/engines/libbluecurve.so 
/emul/ia32-linux/usr/lib/gtk-2.0/2.10.0/engines/libbluecurve.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped
Even though the 32 bit version exists and is available in the library search path used by the dynamic linker, GTK insists on using the 64 bit version. 
Changing LD_LIBRARY_PATH, adding the particular library to LD_PRELOAD, changing -Djava.library.path do not fix the problem. However, a similar easy solution exists - override the search path by setting the GTK_PATH environment variable in a wrapper script for the 32 bit application in question:
export GTK_PATH=/emul/ia32-linux/usr/lib/gtk-2.0
This issue has been reported as Debian bug 490947 and Ubuntu bug 190227. Reports elsewhere linked it to RealPlayer, however this appears to be a problem with the (packaging of) the GTK2 library, so various applications are affected.



About This Blog
2008-12-29T20:05:00.012+01:00
Dear Visitor,
Welcome to my "blog", associated with my professional web site. It is unlikely to become anything resembling a traditional "web-based diary". Rather, I created it as a simple venue for promoting my services by helping other people solve their technical problems and giving away some free advice.
If you are like me, you rely heavily on search engines to find solutions to technical issues and for general troubleshooting inspiration, before attempting a deeper (and more time-intensive) analysis "from first principles". When troubleshooting software, a well-targeted keyword-based search yields fair results most of the time, given a troubleshooter with sufficient expertise. The other times, we learn something new. In a broad sense, we perform successful "applied research", although due to the narrow scope of our problems, there is no room for its results in scientific journals or conference proceedings. Nonetheless, these results carry significant external value measured in time and effort saved by someone else because of their availability. For the troubleshooter himself, writing up a case together with its solution helps gain finer insight into a particular system and commit those heuristics that worked well to memory. Last but not least, it becomes an exercise in successful communication. Making it available to everyone else is then just a matter of courtesy, which in this Internet age may be paid back in the same manner thousandfold.
I hope that you found here what you were looking for. If not, it only takes a few clicks and an email to find out whether I could assist you.
Best regards,
Jan Ploski, plosquare.com

Classic update site	P2 update site
update site = a world-accessible directory on a web server	same as before
update site contains `site.xml`, which contains a list of versioned features installable from this site	update site contains `content.jar` and `artifacts.jar`, which together supersede `site.xml`
subdirectory `features` contains one JAR file per versioned feature, referenced from `site.xml`	same as before, but the references now originate from `content.jar` and `artifacts.jar`
subdirectory `plugins` contains one JAR file per versioned plug-in, referenced from `feature.xml` files contained in feature JARs	subdirectory `plugins` contains one .jar.pack.gz file per versioned plug-in, referenced from `feature.xml` as before, but also from `content.jar` and `artifacts.jar`
	update site (optionally?) contains `digest.zip` - see description below

`JAVA_HOME`	path to where Java 1.5 (or newer) is installed
`launcher`	path to `plugins/org.eclipse.equinox.launcher_1.0.101.R34x_v20080819.jar` from your Eclipse 3.4 (or newer) installation (the version number in the JAR file name may vary)
`output_dir`	path to where the conditioned JAR file should be written - the `plugins` directory of the upgraded update site
`input_file`	path to the input plug-in JAR file

Project	The Perl project containing your scripts and modules.
File to execute	It doesn't matter - choose any script from the project.
Local Host IP	It doesn't matter - enter 127.0.0.1.
Target Host Project Installation Path	Enter the file system path on the remote machine under which your Perl project is deployed (eg. `/home/jpl/modperl`). If you are running Apache on the same machine as Eclipse, enter the path displayed in Project Properties as "Location". If you notice that breakpoints are ignored, this is a key setting to tweak, more on that later.
Port	This is the port on which EPIC is going to listen for a connection from the debugger running on the target Apache host. Enter a port number that is not firewalled (in case of doubt, test connectivity with netcat or the like!). Default is 5000.
Capture Output	It has to be unchecked, as you don't want EPIC to attempt hijacking of STDOUT and STDERR of your scripts!
Create Debug Package	It should be checked on the first invocation (see below). It may be unchecked later on.
Debug Package File Path	Enter the path to a new file that EPIC will create locally when the debugging session is started, for example: `/tmp/epicdb.zip`. This ZIP file will contain a custom version of `perl5db.pl`, which must be used instead of the `Apache/perl5db.pl` in order to support setting breakpoints (it refers to EPIC's `epic_breakpoints.pm` module, also included in the ZIP file). More on that in the next section.

Problem:	When a USB wireless device using the rt73 driver is plugged in under Linux, it appears (e.g. in ifconfig) as rausb0 rather than the more familiar device name wlan0.
Solution:	Add the following line to your /etc/udev/rules.d/70-persistent-net.rules in order to rename the device: ACTION=="add", KERNEL=="rausb0", NAME="wlan0"

/dev/sda1	"Diagnostics" partition; this is a small FAT16 partition preinstalled by Dell. It contains programs for testing memory etc.
/dev/sda2	The Windows XP partition; originally the same block range used to hold a ~10 GB "recovery" partition, which was apparently too small for Vista for its intended purpose.
/dev/sda3	The Windows Vista partition; originally it occupied the rest of disk and had to be shrunk to make room for Linux.
/dev/sda4	The extended partition which holds /dev/sda5, /dev/sda6, /dev/sda7. It is very important to create this partition because any disk can only hold 4 primary partitions. If you forget this little quirk, you will likely end up with large amounts of unpartitioned (that is, unusable) disk space.
/dev/sda5	The Linux boot/root partition.
/dev/sda6	The Linux swap partition.
/dev/sda7	The Linux "data" partition, which in my case holds an encrypted file system for /home and /usr/local (not quite relevant)