tag:blogger.com,1999:blog-1120800400114865482024-03-19T08:16:18.515+01:00plosquare.com blogShort technical articles about troubleshooting and creating software systems by Jan Ploski at plosquare.com.Unknownnoreply@blogger.comBlogger71125tag:blogger.com,1999:blog-112080040011486548.post-89762487107853781412015-06-26T22:43:00.000+02:002015-06-26T22:43:02.227+02:00Unbuffered data fetching from MySQL - mysql_free_result performance issues<p>When you run a SELECT statement through MySQL client library (or some driver implemented on top of it, such as Perl's DBD::mysql or PHP's mysql_query), by default the server buffers the complete query result and returns rows to the client only after the entire query has finished. This is undesirable if the result set is potentially huge and you want to obtain rows as soon as possible. Fortunately, there is a documented solution for this problem - you can configure your client library to use the API function <a href="https://dev.mysql.com/doc/refman/5.6/en/mysql-use-result.html"><code>mysql_use_result</code></a> rather than <a href="https://dev.mysql.com/doc/refman/5.6/en/mysql-store-result.html"><code>mysql_store_result</code></a> (which is equivalent to the --quick option in mysql command-line client).</p>
<p>With that option in place, let's say you wish to implement a sort of client-side LIMIT statement, based on some required client-side filtering of the sorted result set, and discard the rest of the rows as soon as you have found your matches. Unfortunately, your luck ends here because <code>mysql_use_result</code> will not easily satisfy this scenario. When you attempt to finish/release the statement handle after reading a couple rows (as you are required to do, e.g. by calling <code>$sth->finish</code> if you use Perl's DBI), under the hood the function <a href="https://dev.mysql.com/doc/refman/5.6/en/mysql-free-result.html"><code>mysql_free_result</code></a> is called. The documentation makes it appear harmless, saying that it "frees the memory". Apparently, for unknown reasons, this function (or more precisely, <code>cli_flush_use_result</code> called from within it) does a lot more and takes ages to complete given a big result set. Your client may remain blocked for one minute or longer before it is given an opportunity to execute the next statement, all while causing a high CPU usage on the server. You really wanted to simply abort the query, but with MySQL you can't.</p>
<p>This regrettable behavior is even partly documented: <a href="https://dev.mysql.com/doc/refman/5.6/en/c-api-function-overview.html">you must retrieve all the rows even if you determine in mid-retrieval that you've found the information you were looking for</a>, although the reason for it is not explained. The only workaround I found was to close the connection on which the incomplete query was issued and then reconnect immediately. Ironically, this brute-force way of "freeing memory" works instantly in comparison. (Of course, it is not acceptable if you're inside a transaction, which will be lost by dropping the connection.)</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-81331381306215538112014-06-08T14:20:00.001+02:002014-06-08T14:20:08.238+02:00Don't confuse design with art<p>"Design" is a word which seems to be used differently by distinct groups of people. In software engineering, the word has an almost neutral meaning - something along the lines of "the way how the bits and pieces are put together", recognizing that there are many such ways and that they may be told apart based on some definable characteristics. Of course, there is also talk about "good" and "bad" design, implying that in particular contexts certain arrangements may let you reap benefits while others tend to result in drawbacks.</p>
<p>That stronger flavor of "design" is also found in many other disciplines such as law, manufacturing of tools and appliances, usability engineering, game design, and marketing. Here the emphasis is put on intentionally influencing the behavior of users or participants in some favorable way. (The expected favorable effects may occur either for themselves or for someone else, e.g. the party who hired the designer; if a zero-sum game detrimental to the subjects of design is involved, one should more honestly speak of "manipulation" instead). One may also define "design" in a negative way and point out that it comprises all efforts to prevent, well in advance, undesired behaviors or events from occurring during operation of some system. Either way, the first step of design would be to explicitly identify the actors, the utility function that we want to optimize, and the purported causal relationships that connect designed elements to the actors' behavior and their behavior to the optimized function. These causal claims better be supported by empirical data! Thus, design involves meticulous planning and hypothesizing based on experience and facts. It also involves testing to verify the hypotheses. In short, it benefits from a scientific approach.</p>
<p>However, note that there's also a peculiar understanding of "design" proliferated by practitioners of "visual arts" - as found in the phrase "graphics design" and also worryingly often in "web design". This kind of design is indeed closer to "art" and "aesthetics" and satisfying creative whims rather than purposeful control toward achieving well-defined goals. Many a "web designer" is someone interested in (and hopefully proficient in) creating visually pleasing layouts of graphics and type. I do not wish to criticize the merits of such work in general. Printing and typography are both well-established and non-trivial crafts. I want to contrast them with the action-oriented forms of design mentioned earlier. There is not so much action involved in looking at a finely printed book page.</p>
<p>The foremost goal of an artist is not to optimize the performance of a particular task based on hard empirical data or to ensure that someone's sales targets are met. Rather, it is to evoke certain feelings, thoughts and emotions, to impress the viewer in a particular way, to visually communicate their own mood or state of mind, while perhaps following some specific self-consistent style. Even though emotions doubtlessly drive human behavior (possibly much more so than reason, on average), and even though you don't want your audience to experience negative feelings or be annoyed by botched aesthetics of your product, the causal connection of art-induced moods to identifiable sequences of actions and quantitatively measurable goals is notoriously hard to establish. Usually, it is not the "overall impressions", but rather specific actions induced in specific context that matter, the overall impression and "success" emerging as a sum of hundreds of subtle details done right. Not so for many artists who "succeed" in their own eyes, but still fail miserably on any other objective account, including on account of "design" in the above elucidated stricter senses of the word. If artists called themselves doctors, most of us would prefer to call them quacks.</p>
<p>In conclusion, when you decide to obtain advice from, or services of, a "designer", do pay attention to what they perceive as their job definition, especially to the extent in which they recognize the mundane, utilitarian aspects. Failing that, you might find yourself in place of an art patron rather than recipient of effective business services. Your return on investment may then amount to subjective satisfaction predicated on your own suspension of disbelief and submission to the artist's superior sense of authority. <a href="http://www.youtube.com/watch?v=K2ypKk57zl4" rel="nofollow">Don't let them fool you.</a></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-13933836241961915262014-05-21T19:05:00.002+02:002014-05-21T19:05:50.456+02:00Chrome 35 on Linux - Java plug-in not working<p>Today Google demonstrates to us how not to update critical platform software: the latest stable version 35.0.1916.114 contains "a number of fixes and improvements", one of them being <strong>dropped support for Oracle's Java plug-in on Linux</strong>, as a consequence of discontinuing support for the prerequisite NPAPI (one wonders whether it has anything to do with the ongoing <a href="http://en.wikipedia.org/wiki/Oracle_v._Google">Google vs. Oracle case</a>).</p>
<p>After installing the new version there is absolutely no notification to the user that this functionality, which is essential for some applications (e.g. online banking and some enterprise applications), is no longer available. Although the plug-in symlink is present in the correct location and it wouldn't be difficult to alert about incompatibility, it simply doesn't appear in the about:plugins list. For sure hundreds of hours will be wasted in sum by users trying to troubleshoot this issue on their and following outdated instructions. As of 2014-05-21, there is no replacement Java plug-in for Chrome 35 on Linux available.<p>
<p>For more information see this <a href="https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/xEbgvWE7wMk" rel="nofollow">discussion thread</a>.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-40353953116365183642014-05-10T20:23:00.004+02:002014-05-10T20:23:34.570+02:00Solution for mount fails with "No such file or directory"If you find yourself stuck in an initramfs Busybox shell, trying to manually mount the root filesystem (using a command such as mount /dev/mapper/yourrootfs /root) and getting a confusing "No such file or directory" message in response, then check whether the appropriate kernel module for your filesystem type is loaded (i.e. run "lsmod"). Something as simple as "modprobe ext2" before the mount command might be the entire solution you need (it was for me today, in context of an interrupted Debian dist-upgrade from squeeze to wheezy)!Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-112080040011486548.post-75639744462262235172014-04-13T16:45:00.002+02:002014-04-13T16:45:29.667+02:00Don't optimize too early, don't optimize too late<p>Software engineering, just like any kind of engineering, is a discipline which involves many informed, planned compromises—a surprising amount of "economizing".</p>
<p>The task of optimizing software—that is, minimizing its resource requirements within an effort budget allocated toward this purpose—is a classical example in which economic realities of software development were early recognized by the pioneers our our discipline. Thus we have Knuth's adage that "premature optimization is the root of all evil". Reasonably, as performance engineers we should focus our efforts on those areas that (after measurement) consume most resources, rather than waste time on following personal hunches about imagined "best practices". One might describe this process as "top-down", or strategic, optimization. First use measurement tools to identify the critical areas and the big decisions that affect performance (such as algothmic complexity), then fix the identified problems.</p>
<p>The top-down optimization approach works very well in most cases, but sometimes it fails. That is because some performance problems are incremental and combinatorial in nature. Rather than being located in only a few critical areas of code, a low-hanging fruit for a trained performance engineer, they turn out as pervasive issues - where performance suffers "death by a thousand cuts", not due to some grand ommissions, but due to universal cumulative neglect.</p>
<p>Recognition that such <em>systemic</em> problems do exist and that they are a lot more difficult to deal with than local bottlenecks should make us think again about the dangers of the popular, easy-going "make it work, make it right, make it fast" approach to software engineering (or should we say, software prototyping?)</p>
<p>In other industries serious opposite viewpoints already exist that treat quality as an <em>emergent</em> property of products. Historically, mastery in any craft has always involved paying relentless attention to every detail <em>throughout</em> the process of crafting and construction. as opposed to post hoc economizing and extinguishing of only the most critical problems. In that respect, recall-and-repair, also a form of "late improvement", is a practice more related to modern mass production. Not by coincidence, mass production works best where discerning patrons or purchasers are also replaced by relatively hapless consumers onto whom manufacturers may shift some of their production risks in exchange for a lower price (or higher profit). High-end, highly customized products seldom benefit from such conditions.</p>
<p>Incidentally, paying attention to detail is also something that is psychologically natural for individuals who understand and love developing software, and a source of personal pride and motivation (sometimes it also degenerates into hair-splitting; as always, you can have too much of a good thing). At heart of what I call bottom-up optimization approaches is a conviction that a flawless final product can only be a result of combining flawless components withing a flawless process, a core belief that superior quality pays off in a myriad ways, not excluding immaterial rewards from feeling of accomplishment. It is a conviction which most software engineers accept quite readily when it comes to considering functional correctness of their products. Nobody would use a buggy standard library or consider fixing its bugs as either "premature" or "evil". However, possibly due to overemphasizing and misinterpretation of Knuth's wise statements by some lesser capable minds, the same level of attention to detail is rarely spent on designing software performance (except in systems in which hardware costs cannot be shifted onto users or poor performance translates into incorrect execution; or in systems with such great competition that inadequate performance translates into lost sales or operational losses on hardware).</p>
<p>So what should we make of those divergent views on early and late, bottom-up and top-down optimizations? I think the key to striking the correct balance ultimately lies in understanding how value is created and destroyed within the software development process. It is a strategic mistake to focus your optimization efforts on those areas which are easy to measure (e.g. late profiling) while neglecting the difficult ones. On the other hand, it is a tactical mistake to waste effort on improvements of provably little consequence, especially while disregarding clear opportunity costs. But, as in any discipline, it takes some years of practice to reliably distinguish which kind of those mistakes you are making (of which even experts may disagree). Always optimize your bottom line—but understand first how many numbers stand above it and how they are connected.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-80559491389351667272014-01-12T16:40:00.000+01:002014-01-12T16:40:36.642+01:00Some functional programming tricks and gotchas<p>I took some quick notes comparing procedural and functional programming approaches after listening to <a href="http://www.youtube.com/watch?v=m3svKOdZijA">Brian Lonsdorf's talk about Underscore.js</a>. Currently, I'm not very enthusiastic about pure functional programming. I'm leaning towards the mainstream opinion which banishes it into the role of syntactic sugar in largely procedural and object-oriented libraries, given the huge amount of existing non-functional code out there.</p>
<p>In procedural programming you invoke functions with arguments. In functional programming you apply functions to arguments (or more generally, to some data structures).</p>
<p>In procedural programming there is a strict time and space separation between defining functions and using (invoking) them. In functional programming it's very common to not just pass around functions as arguments or return values, but also to define new functions by invoking other functions. Established patterns for function generation exist.</p>
<p><strong>Perhaps the most strikingly confusing assumption when looking at FP code through eyes of a procedural programmer is that an expression f(something) represents a runtime invocation of f, which returns some data structure.</strong> This assumption, universally true in procedural world, does not hold in functional programming where such expressions commonly represent functions (and are possibly even subject to static optimizations by compilers). Unfortunately, no naming conventions seem to exist that would signify whether a function returns another function or data structure (which seems to me like a rather crucial distinction for aiding code comprehension). <strong>Tip: if you see the expression f(x) in functional programming code, try comprehending the whole (and not just f) as a function name.</strong></p>
<p>In procedural programming it's usual that functions are uniquely named and static (they aren't created or destroyed at runtime). In functional programming anonymous temporary functions are common.</p>
<p>One common pattern found in functional programming is creating a new function based on another by fixing the first few arguments to constant values ("partial function application"). Argument lists are intentionally designed to allow this. The rule of thumb is to place the more variable arguments later in an argument list. More specifically, arguments that represent data to be processed should appear after such that represent functions to be applied to the data. This lets you specialize more general functions by replacing their function arguments with more concrete functions of your own.</p>
<p>A programming technique to support the argument fixing trick is known as "currying". Curry is an operation that can be (mechanically) applied to any function with multiple arguments. It returns a semantically equivalent function of the first argument, which in turn returns a function of the second argument, and so on. For example, currying a function of three arguments f(x,y,z) yields f', which you can then apply as follows: f'(x)(y)(z). You can now used this f' to easily create a function f(X,Y,z), with X and Y being fixed constants - simply by invoking f'(X)(Y).</p>
<p>Another common pattern for creating new functions from others is composition. You use composition if you want a function which applies some function to the result to another (a nested function invocation, in procedural terms). More generally, you can compose n functions (i.e. multiple levels of nested invocation). compose(f,g,h) will give you a new function which will produce the same result as f(g(h(...))) when applied.</p>
<p>The 'map' function, commonly found in functional programming languages as an idiom for processing a list of elements by applying some argument function, can be usefully generalized by allowing it to process not just lists but other also other objects known as "functors". To be supported as argument for map, such an object must provide an own implementation of 'map', which is then used in place of the default one to return a list of elements. Useful idioms that can be implemented this way include Maybe(x) and Either(x,y) of Haskell. The Maybe object applies the mapped function to its argument only if it is not null. The Either object applies to the mapped function to one of its arguments (allowing that one of them is not null).</p>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-37344106754149108862013-11-23T23:19:00.000+01:002013-11-23T23:19:48.866+01:00MySQL 5.6 memcached plugin performance benchmarks<p>MySQL 5.6 introduces a new feature - you can talk to the InnoDB enginge using the memcached protocol, which is supposed to be faster for simple get/set operations based on a key. For an introduction, refer to this <a href="http://chipersoft.com/p/MySQL-via-Memcache/">MySQL via Memcache tutorial</a> and the <a href="http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached.html">MySQL manual</a>. (The basic idea is that you create a special database in which you describe your tables that should be "exported" to be accessible through the memcached plugin and then enable the plugin itself.)</p>
<p>I didn't find any ballpark figures about how much improvement the new memcached backend might actually offer for applications, so I ran a few benchmarks myself. I compared it to a straightforward way of using an InnoDB table as a key-value store via SQL (either with DELETE/INSERT or REPLACE) and also (just for fun) against a real memcached backend (which of course unlike MySQL offers no persistence). The versions I used specifically were InnoDB 5.6.14 and memcached 1.4.5. The benchmark client wrote about 270000 key-value pairs in random order to the test.demo_test table shipped with the plugin as an example. Each key consisted of 10 and each value of 176 bytes. I also ran another test with 4 such clients running in parallel (on a quad-core AMD Phenom 9650 machine).</p>
<p>I also tested two ways of setting the keys - either with one operation per key or with a batch operation set_multi (this is something offered by the Cache::Memcached::Fast client, not in the memcached protocol, as far as I'm aware).</p>
<p>Note that the DELETE/INSERT approach is not recommended as it is not only slow, but also causes errors and/or deadlocked transactions on collisions (I didn't bother implementing transaction retry).</p>
<TABLE FRAME=VOID CELLSPACING=0 COLS=7 RULES=NONE BORDER=0>
<COLGROUP><COL WIDTH=145><COL WIDTH=83><COL WIDTH=89><COL WIDTH=92><COL WIDTH=95><COL WIDTH=79><COL WIDTH=176></COLGROUP>
<TBODY>
<TR>
<TD HEIGHT=17 ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT><BR></TD>
<TD COLSPAN=2 ALIGN=CENTER VALIGN=MIDDLE>1 client</TD>
<TD COLSPAN=3 ALIGN=CENTER VALIGN=MIDDLE>4 clients</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT><B>WRITE</B></TD>
<TD ALIGN=RIGHT>ms/op</TD>
<TD ALIGN=RIGHT>ops/client/s</TD>
<TD ALIGN=RIGHT>ms/op</TD>
<TD ALIGN=RIGHT>ops/client/s</TD>
<TD ALIGN=RIGHT>total ops/s</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>DELETE+INSERT</TD>
<TD ALIGN=RIGHT SDVAL="0.47875" SDNUM="1033;">0.47875</TD>
<TD ALIGN=RIGHT SDVAL="2088.772845953" SDNUM="1033;0;0">2089</TD>
<TD ALIGN=RIGHT SDVAL="0.783235329373401" SDNUM="1033;0;0.00000">0.78324</TD>
<TD ALIGN=RIGHT SDVAL="1276.75548139537" SDNUM="1033;0;0">1277</TD>
<TD ALIGN=RIGHT SDVAL="5107.02192558149" SDNUM="1033;0;0">5107</TD>
<TD ALIGN=LEFT>(not safe)</TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>REPLACE</TD>
<TD ALIGN=RIGHT SDVAL="0.28412" SDNUM="1033;">0.28412</TD>
<TD ALIGN=RIGHT SDVAL="3519.6395889061" SDNUM="1033;0;0">3520</TD>
<TD ALIGN=RIGHT SDVAL="0.555974761974184" SDNUM="1033;0;0.00000">0.55597</TD>
<TD ALIGN=RIGHT SDVAL="1798.64279531169" SDNUM="1033;0;0">1799</TD>
<TD ALIGN=RIGHT SDVAL="7194.57118124677" SDNUM="1033;0;0">7195</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>InnoDB set</TD>
<TD ALIGN=RIGHT SDVAL="0.26271" SDNUM="1033;">0.26271</TD>
<TD ALIGN=RIGHT SDVAL="3806.47862662251" SDNUM="1033;0;0">3806</TD>
<TD ALIGN=RIGHT SDVAL="0.513042672692542" SDNUM="1033;0;0.00000">0.51304</TD>
<TD ALIGN=RIGHT SDVAL="1949.15560288936" SDNUM="1033;0;0">1949</TD>
<TD ALIGN=RIGHT SDVAL="7796.62241155744" SDNUM="1033;0;0">7797</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>InnoDB set_multi</TD>
<TD ALIGN=RIGHT SDVAL="0.120204225589037" SDNUM="1033;0;0.00000">0.12020</TD>
<TD ALIGN=RIGHT SDVAL="8319.17509638033" SDNUM="1033;0;0">8319</TD>
<TD ALIGN=RIGHT SDVAL="0.474549357414406" SDNUM="1033;0;0.00000">0.47455</TD>
<TD ALIGN=RIGHT SDVAL="2107.26236244113" SDNUM="1033;0;0">2107</TD>
<TD ALIGN=RIGHT SDVAL="8429.04944976451" SDNUM="1033;0;0">8429</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>memcached set</TD>
<TD ALIGN=RIGHT SDVAL="0.12067843431908" SDNUM="1033;0;0.00000">0.12068</TD>
<TD ALIGN=RIGHT SDVAL="8286.48470327307" SDNUM="1033;0;0">8286</TD>
<TD ALIGN=RIGHT SDVAL="0.112353995880574" SDNUM="1033;0;0.00000">0.11235</TD>
<TD ALIGN=RIGHT SDVAL="8900.44000805224" SDNUM="1033;0;0">8900</TD>
<TD ALIGN=RIGHT SDVAL="35601.760032209" SDNUM="1033;0;0">35602</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>memcached set_multi</TD>
<TD ALIGN=RIGHT SDVAL="0.0403774059578413" SDNUM="1033;0;0.00000">0.04038</TD>
<TD ALIGN=RIGHT SDVAL="24766.3260251071" SDNUM="1033;0;0">24766</TD>
<TD ALIGN=RIGHT SDVAL="0.143840798391106" SDNUM="1033;0;0.00000">0.14384</TD>
<TD ALIGN=RIGHT SDVAL="6952.13048860434" SDNUM="1033;0;0">6952</TD>
<TD ALIGN=RIGHT SDVAL="27808.5219544174" SDNUM="1033;0;0">27809</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT SDNUM="1033;0;0"><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT><B>READ</B></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT SDNUM="1033;0;0"><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>SELECT</TD>
<TD ALIGN=RIGHT SDVAL="0.16261" SDNUM="1033;">0.16261</TD>
<TD ALIGN=RIGHT SDVAL="6149.6832913105" SDNUM="1033;0;0">6150</TD>
<TD ALIGN=RIGHT SDVAL="0.13341904412122" SDNUM="1033;0;0.00000">0.13342</TD>
<TD ALIGN=RIGHT SDVAL="7495.18186542719" SDNUM="1033;0;0">7495</TD>
<TD ALIGN=RIGHT SDVAL="29980.7274617087" SDNUM="1033;0;0">29981</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>InnoDB get</TD>
<TD ALIGN=RIGHT SDVAL="0.099201552821732" SDNUM="1033;0;0.00000">0.09920</TD>
<TD ALIGN=RIGHT SDVAL="10080.487366937" SDNUM="1033;0;0">10080</TD>
<TD ALIGN=RIGHT SDVAL="0.0752699392179072" SDNUM="1033;0;0.00000">0.07527</TD>
<TD ALIGN=RIGHT SDVAL="13285.5162418159" SDNUM="1033;0;0">13286</TD>
<TD ALIGN=RIGHT SDVAL="53142.0649672635" SDNUM="1033;0;0">53142</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
<TR>
<TD HEIGHT=17 ALIGN=LEFT>memcached get</TD>
<TD ALIGN=RIGHT SDVAL="0.0613956630145077" SDNUM="1033;0;0.00000">0.06140</TD>
<TD ALIGN=RIGHT SDVAL="16287.7954386404" SDNUM="1033;0;0">16288</TD>
<TD ALIGN=RIGHT SDVAL="0.0344708070262006" SDNUM="1033;0;0.00000">0.03447</TD>
<TD ALIGN=RIGHT SDVAL="29010.0547759128" SDNUM="1033;0;0">29010</TD>
<TD ALIGN=RIGHT SDVAL="116040.219103651" SDNUM="1033;0;0">116040</TD>
<TD ALIGN=LEFT><BR></TD>
</TR>
</TBODY>
</TABLE>
<p>My conclusion is that the new memcached backend can improve MySQL read performance quite a lot (77% more reads per second) and improve write performance a little (8%) compared to a pre-5.6 approach using SELECT/REPLACE. The percentual speedup for reads may seem big, but it is negligible in absolute terms (retrieving single rows through key lookup was already very fast). So switching to the new backend would only make sense for applications that experience really heavy read loads; it won't improve response time of an average application which is mostly sitting around idle. And of course if need no persistence nor SQL access to your cache tables (which sounds nice for performing bulk updates/deletes as may be), then the good old memcached is a clear winner.</p>
<p>Download my benchmark script and the results in spreadsheet format:</p>
<ul>
<li><a href="http://www.plosquare.com/sites/default/files/download/innodb_memcached.xls">MySQL 5.6 vs. memcached performance (Excel spreadsheet)</a></li>
<li><a href="http://www.plosquare.com/sites/default/files/download/innodb_memcached.pl">MySQL 5.6 vs. memcached performance (test driver)</a></li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-1618711347433272662013-02-08T17:14:00.001+01:002013-02-08T17:14:42.703+01:00Solution for GSS-API major_status:00090000, minor_status:861b6d0c<b>Problem:</b> You are trying to configure mod_auth_kerb to work with Active Directory. You have created the technical user account "yourserver" for HTTP service in AD and associated it with the Kerberos SPN HTTP/your.server.com@YOURDOMAIN.COM using ktpass.exe. When accessing the page which requires Kerberos auth with IE, you see the following error messages in Apache error log (after raising LogLevel to debug):
<br />
<pre>[client nnn.nnn.nnn.nnn] Warning: received token seems to be NTLM, which isn't supported by the Kerberos module. Check your IE configuration.
src/mod_auth_kerb.c(1101): [client nnn.nnn.nnn.nnn] GSS-API major_status:00090000, minor_status:861b6d0c
[client nnn.nnn.nnn.nnn] gss_accept_sec_context() failed: A token was invalid (, Unknown code)</pre>
You have checked your IE (or Firefox) configuration and are pretty sure that the browser should be sending a Kerberos ticket instead of attempting NTLM authentication.<br />
<br />
<b>Solution:</b> First apply some more diagnostics:<br />
<ol>
<li>Check if the command <pre>kvno HTTP/your.server.com@YOURDOMAIN.COM</pre>
gives you this message: "HTTP/your.server.com@YOURDOMAIN.COM: Server not found in Kerberos database while getting credentials". If yes, you likely have the problem described here.</li>
<li>On the Windows AD server check the output of <code>setspn -l yourserver</code>. Does it appear like so?
<pre>Registered ServicePrincipalNames for CN=yoursever,OU=Service Accounts,OU=Accounts,DC=yourdomain,DC=com:
HTTP/your.server.com<span style="color: red;">@YOURDOMAIN.COM</span></pre>
If yes, then you likely have the problem described here. The domain suffix highlighted in red should <i>not</i> appear in the command's output in a correct configuration.</li>
</ol>
To fix the incorrect SPN association, use the following commands on AD server:
<br />
<pre>setspn -d HTTP/your.server.com@YOURDOMAIN.COM yourserver
setspn -A HTTP/your.server.com yourserver
</pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-15051094902551080222013-02-08T16:45:00.003+01:002013-02-08T16:53:31.075+01:00Solution for GSS-API major_status:000d0000, minor_status:96c73ae6<b>Problem:</b> You are trying to configure mod_auth_kerb. When you access a page that requires Kerberos auth with IE, a popup asking for password appears. In Apache error log (after raising LogLevel to debug), you can see the following messages appearing:
<br />
<pre>[debug] src/mod_auth_kerb.c(1101): [client nnn.nnn.nnn.nnn] GSS-API major_status:000d0000, minor_status:96c73ae6
[error] [client nnn.nnn.nnn.nnn] gss_accept_sec_context() failed: Miscellaneous failure (, Unknown code)
</pre>
<b>Solution:</b> Check that the kvno (key version number) stored in your keytab matches the kvno reported by the kvno command. You can view kvno from keytab using <br />
<pre>klist -Kekt /path/to/keytab</pre>
Compare this against <br />
<pre>kvno HTTP/your.server.com</pre>
The kvno command only works after a successful authentication with kinit.
You can recreate your keytab with the expected kvno (in the example below: 3) using ktutil:
<pre>add_entry -password -p HTTP/your.server.com@YOURDOMAIN.COM -k 3 -e RC4-HMAC
wkt /path/to/newkeytab
</pre>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-42051182364910411562013-01-16T21:31:00.000+01:002013-01-16T21:31:11.444+01:00Solution for "Key table entry not found while getting initial credentials"<b>Problem (with Kerberos on Debian stable):</b><br />
<pre>kinit -V -k -t /tmp/keytab HTTP/somehost@LOCALDOMAIN
kinit: Key table entry not found while getting initial credentials
</pre>
However <code>klist -k -t /tmp/keytab</code> works and correctly displays HTTP/somehost@LOCALDOMAIN as present under Principals. The keytab entry was previously created with using <code>addent -password -p HTTP/somehost@LOCALDOMAIN -k 1 -e rc4-hmac<code>.
</code></code><br />
<code><code><b>Solution:</b></code></code><br />
<code><code>
Had to adjust /etc/krb5.conf:<br />
<pre>default_tkt_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5
default_tgs_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5</pre>
<a href="http://www.novell.com/support/kb/doc.php?id=7005039">Source link</a></code></code>Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-112080040011486548.post-90464869179005517352012-11-11T17:16:00.002+01:002012-11-11T17:16:33.496+01:00Running OQL queries from command line<p>OQL is a specialized query language for memory profiling of Java heap dumps. It is implemented by tools such as jvisualvm and jhat, which ship with current versions of JDK. Unfortunately, neither of these tools provides a simple command-line interface out of the box to integrate them into your own development environment and support real-life scripting. Rather, each of them gives you a kludgy and insufficient GUI.</p>
<p>However, it is easy simply to feed queries to the web server provided by jhat. You can download the <a href="http://www.plosquare.com/sites/default/files/download/oql.pl">OQL command line Perl script</a> to do just that (<a href="http://www.plosquare.com/sites/default/files/download/oql.html">see documentation</a>).</p>
<p>A few notes from my playing around with OQL:</p>
<ul>
<li>The ability to define your own Javascript functions is very useful, go for it!</li>
<li>Unfortunately, some of the handy built-in functions provided by jvisualvm are absent from jhat (need to define your own).</li>
<li>To obtain object IDs that can be passed to heap.findObject you must use the built-in function objectid(objectReference). The identifiers displayed by jvisualvm that look like <i>#nnn</i> are <em>not</em> objectids (that would be too easy, eh? I don't see any way to obtain them using jvisualvm GUI, have to use OQL).</li>
<li>To display the entire content of a char[] array (which jvisualvm cuts off in a useless way), you must use an OQL expression like objectReference.toString().</li>
<li>If OQL scares you, consider buying a commercial profiler.</li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-71177320956655924342012-11-10T16:47:00.000+01:002012-11-10T16:49:18.661+01:00Software as a process<p>I have always had a (largely concealed) aversion to the term "software architecture". Indeed I even used to think: the less architecture you had, the better. "Software architecture" seemed to me as an euphemism for "premature constraints" that limited the directions into which your software might evolve; the new clothes on the widely disregarded "waterfall" model. Furthermore, before the rather pompous sounding ideas of "software architecture" were invented (and then standardized by ISO commitees), we had already had notions of good "software design" covering about the same themes.</p>
<p>I was also suspicious about comparing software to other disciplines such as civil engineering where physical structures are erected. Another analogy, city planning, commonly invoked by those colleagues who prefer to call themselves "enterprise architects" was somewhat less alien, but still not quite appealing for some reason.</p>
<p>I used to attribute that apprehension to my perceived malleability of software, the relative ease of changing it, adding to it, reconfiguring it. I still think that is a valid reason to object to analogies that also let us think about software lifecycle as "build (or buy)", "use", "replace" phases and that inescapably guide our thoughts toward the established notion of licensed "software products". But I feel there is also a nimbler mental notion which neatly explains my discomfort with those mainstream ideas: software as a process.</p>
<p>Processes - as in "business processes" or "industrial processes" - are a way of modelling the world in terms of events, changes and dependencies. Good things happen. Bad things happen. Resources are consumed and produced. Information starts and stops flowing between entities. Entities come into existence. Entitites disappear. Databases grow. Users request services. Customers are won and lost. Cash flows. The world is in motion. Careful, continuous process control at all levels is what we need to avoid costly mistakes.</p>
<p>Our role as software developers is often not to invent some sort of "best practice architecture" and then naively stick to it, not even to bring a specified product to market. Instead, it is to react to environmental changes and guide our software through a path of constant change. It is to ensure that we maintain control of that what we once built - and keep extending, adapting and improving - and to design our software in such a way that the broader processes in which it participates are supported rather than hindered. It is to anticipate scenarios with risks and opportunities and to make them less or more probable to fulfill business objectives. Such scenarios always concern both the currnet use and the future evolution of the systems we create.</p>
<p>By the way, the distinction between "programming in the large" and "programming in the small" is FUD. When viewed through the lenses of process control, both areas indeed appear much the same, unified even. The structure of our engineering problems is more akin to a fractal or a piece of music: the same (few) recurring themes and prinicipal rules of composition appear both in the large and in the small. However, few people flip-flop between work at different scales, which leads to an unfortunate multiplication of different names for similar concepts.</p>
<p>Today I am relieved to see that for several years I haven't been the dumb one who was stubbornly rejecting the grand concepts of software architecture and marketing myself as some awkward combination of "developer" and "systems administrator" (a jack of all trades...) at my own peril. The ideas outlined above now seem to be gaining increasing popularity among highly respected practitioners charged with maintaining big software systems in big (Internet) companies. Lo and behold, there even seems to be a catchy buzzword: "devops", a mix of the traditional roles "developement" and "operations".</p>
<p>By the way, I am not opposed to the job role of "software architect", if you interpret it as someone who is hired by a client to keep watch against imagination of inexperienced software developers (although "software process manager", "master devop" or simply "chief programmer/designer" might still be more accurate descriptions of the job).</p>
<p>Last, but not least: YMMV. The world of software development is diverse that it is likely that no single mental model will fit all imaginable contexts; or that the signle grand unified model may prove too abstract to be immediately useful. It makes quite a practical difference if you are designing shrink-wrapped software products for consumers or planning or maintaining installations of enterprise or Internet SaaS applications. Pick your specific models wisely, so as to benefit your own work.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-14366971043243128302012-10-30T23:05:00.000+01:002014-03-31T22:08:46.189+02:00Uncorrelated subqueries treated as DEPENDENT by MySQL<p>Are you using the operator IN with subqueries in MySQL and wondering about poor performance? Have you tried EXPLAINing your queries and noticed that they are reported as "DEPENDENT SUBQUERY" despite your best intent to use an uncorrelated subquery? If so, read on:</p>
<p>This post describes a MySQL bug (some call it a feature) which has
been open for many years and according to official documentation still
exists in version 5.6 (note that I only checked 5.0.x and 5.1.x myself
and didn't bother to look for info about 6.0, but I have no high
hopes based on the bug's age and character).</p>
<p>Consider the following two SQL statements:</p>
<ol>
<li><pre>SELECT id FROM t1 WHERE id IN (1, 2, 3)</pre>
</li>
<li><pre>SELECT id FROM t1 WHERE id IN (SELECT id FROM t2)</pre>
</li>
</ol>
<p>In MySQL they have very different performance characteristics:
the first one will work fast as expected, the second one will actually
cause index lookups to be executed repeatedly for each row from t1
rather than just once for the entire statement. Why?</p>
<p>Blame the "optimizer", as explained in the <a href="http://dev.mysql.com/doc/refman/5.6/en/subquery-restrictions.html">manual</a>. Our fine uncorrelated query becomes forcingly "optimized" into an EXISTS query and thus correlated (rougly speaking; it's a bit more complicated, but you can view the result by using EXPLAIN EXTENDED followed by SHOW WARNINGS). MySQL of course must handle correlated queries per row, leading to disastrous algorithmic performance in what one would assume is a very common case.</p>
<p>I have no clever workaround - you can't "convince" the MySQL optimizer to behave differently. What you can do is use temporary tables to simulate what you really want to happen. But if you do, beware that you may only refer to a temporary table once in a SELECT statement due to <a href="http://dev.mysql.com/doc/refman/5.6/en/temporary-table-problems.html">another hugely annoying limitation</a>.</p>
<p>For further information see:</p>
<ul>
<li><a href="http://dev.mysql.com/doc/refman/5.6/en/subquery-restrictions.html">Subquery restrictions according to MySQL manual</a></li>
<li><a href="http://bugs.mysql.com/bug.php?id=9090">Optimization Problem with a subquery in an IN Operation</a></li>
<li><a href="http://stackoverflow.com/questions/3597021/mysql-where-in-subquery-runs-forever">MySQL WHERE IN subquery runs forever</a></li>
<li><a href="http://optimize-this.blogspot.com/2010/06/problems-with-subquery-transformations.html">Problems with subquery transformations</a></li>
<li>Update 20140331: <a href="http://oysteing.blogspot.de/2012/07/from-months-to-seconds-with-subquery.html">Improvements in MySQL 5.6 - Øystein Grøvlen explains subquery materialization</a>, also see <a href="https://archive.fosdem.org/2013/schedule/event/new_opt_mysql/attachments/slides/288/export/events/attachments/new_opt_mysql/slides/288/fosdem2013.pdf">his slides</a>, starting from page 15.</li>
</ul>
<p>P.S. If you found a MySQL version that works in a smarter way, don't hesitate to drop a comment.</p>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-112080040011486548.post-65024827362264598992012-03-04T11:01:00.002+01:002012-03-04T11:15:23.102+01:00Adding attributes to SOAP-Lite envelope<p>There is a common theme in developing software while having to use badly designed libraries: you know exactly what you want to achieve, and are not asking for much, but a particular library prevents you from progressing. Without any doubt, SOAP::Lite in Perl is this kind of a library. Suppose we want to perform some simple modification of the generated XML SOAP message, such as adding another attribute to the Envelope element. This shouldn't be hard at all given the purpose of the library. Yet it turns out otherwise. Is there an officially blessed way of doing it with SOAP::Lite? I don't know. But there is this hack:</p><pre>
$envelope_orig = \&SOAP::Serializer::envelope;
*SOAP::Serializer::envelope = sub {
my $str = $envelope_orig->(@_);
$str =~ s{xmlns:xsi=}{xsi:schemaLocation="http://whatever" xmlns:xsi=}s;
return $str;
}
</pre><p>By replacing the SOAP::Serialize::envelope subroutine you can alter the resulting XML message in whatever way you wish. This is working around the API, a potentially dangerous thing to do in general (though in this particular case, the danger of SOAP::Serialize::envelope being removed in a next version is slim). Use at your own risk.</p>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-112080040011486548.post-20132922283622491882011-09-11T20:20:00.009+02:002011-09-11T21:55:57.921+02:00On errors masked and revealed<p>In a <a href="http://edge.org/conversation/the-marvels-and-flaws-of-intuitive-thinking">recent talk</a> professor Daniel Kahneman looks back at his research in cognitive psychology. In a side remark he notes that errors are "much more diagnostic about the mechanisms, so there might be different mechanisms for doing the same thing perfectly, and you're not going to be able to distinguish the mechanism by observing it". The approach of focusing on errors (or anomalies) in trying to accurately model something is of course a long-standing one. For example, we have the study of anomalies in medicine; natural experiments in social sciences; and of course the practice of intentionally breaking and perturbing things in about any other scientific discipline, including software engineering.</p><p>However, one of the striking characteristics of software engineers (perhaps shared with other kinds of designers, too) is that, unlike those scientists who attempt to peek under the hood of nature's work, we are obsessed with the idea of hiding the "implementation details" of our own creations - that is, abstracting away the mechanisms by which they operate to the greatest possible extent at all conceivable levels. I think that the driving force behind this is not so much the traditionally touted flexibility of exchanging implementations (which, to be honest, seldom needs to be done) and interoperability. Rather, it is our desire to simplify our own mental models, to reduce the cognitive load of having to think about (and worse, test) too many elements and interactions at once. Maybe so much so as to shift some of the processing from System 2 to System 1, in Kahneman's terms. We strive to reduce the probability of mistakes - which on the economic level also means reducing development and maintenance costs for our clients.</p><p>When software systems fail, the tables may be turned on us - the hidden mechanisms then become apparent, and the usually blessed lack of familiarity with the implementation details becomes a curse. We are then forced to learn things about our systems that we have never bothered to understand. In more extreme cases, this aspect may even represent a significant liability and risk to the maintainer, if contractual obligations demand quick, correct reactions.</p><p>I think a few conclusions may be drawn from the above:</p><ol><li>Less intricate, "uglier" designs that reveal more of the underlying mechanisms, contain fewer layers of indirection, and put a smaller demands on supporting technology and tools might be actually preferable for economic reasons - more attractive from the point of view of contingency planning (if a total or maximum time-to-repair is a concern).</li><li>The skills of a software developer and those of a maintainer or troubleshooter are intricately linked, but nevertheless quite distinct. Great software developers know which internal aspects of (their) software to hide and which to reveal to make its normal operations easy to grasp and the rare breakdown possible to handle. Great troubleshooters know how to reveal those aspects that were intentionally hidden by the (possibly not-so-great) developers in order to understand the causal connections and required interventions. Ideally, they also have an extensive knowledge of the under-the-hood mechanisms of a particular software product or system which they maintain.</li><li>The skill set of a software developer is rightfully viewed as a superset of (and thus more valuable than) that of a troubleshooter or system administrator: a developer typically not only has to troubleshoot an analyze other developers' software because of the interfacing and teamwork requirements, but he also has to anticipate and support the future troubleshooting scenarios that will occur with respect to his own software. On the other hand, he can usually avoid accumulating truly significant amounts of factual knowledge about particular products or operating environments and rely on more general procedural knowledge.</li></ol><p>I don't know about others, but I do seem to spend most of my software development time thinking about errors and mistakes - real and possible, static, dynamic, happening inside of my code and in the users' world around it.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-1691321347501570292011-07-01T18:29:00.004+02:002011-07-01T18:36:52.957+02:00Fix for "No such file or directory" while trying to mount --bind<p><strong>Problem:</strong> The command <code>mount --bind /mnt/data/usr /usr</code> in Linux fails with "No such file directory". <code>/mnt/data</code> is a mount point of a dm-crypt device. strace shows ENOENT for the mount syscall, no other clues. Variants that don't refer to <code>/usr</code> (for example, to <code>/usr2</code> instead), do work as expected. The same error occurs after and during boot process, so it's not explainable by used files.</p><p><strong>Solution:</strong> The problem disappeared after upgrading the kernel from 2.6.34 to 2.6.38.8.</p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-112080040011486548.post-67249853430956308572011-06-26T01:42:00.006+02:002011-06-26T02:24:28.722+02:00Fix for random frequent freezes in GTK applications (Debian 'testing')<p>Sometimes software configuration problems are solved thanks to a deep insight into how things work and methodical experimentation. At other times they are solved through a shotgun approach in which hunches, sheer luck and brute force are combined to get rid of a problem without ever truly understanding its cause. This leaves no useful information behind for <em>avoiding</em> the problem in the future, but maybe we don't care. The following describes such a situation.</p>
<p>After upgrading some packages, notably xorg, in Debian ('testing'), most GTK applications (e.g. Firefox, SeaMonkey, Eclipse, ...), started crashing intermittently, with just the infamous unresponsive UI symptom (aka freeze, hang). After some Googling I noticed that adding the <code>--sync</code> command line option alleviated the problem, and for a period of time I just used this as a crude but effective workaround. I got motivated to look for a real fix when I ran into the same symptoms while trying to run a hosted Eclipse workbench (a PDE JUnit test suite). Here the hosted workbench would freeze almost immediately after startup (the window rendered, but without the menu bar, after which no more repaint events were processed). It would happen always without the <code>--sync</code> option, but also very often despite it being supplied. I observed in the JDT debugger that the lock-up would always occur when the main thread was callnig a GTK/GDK function - not always the same function (sometimes <code>gdk_flush</code>, sometimes <code>gdk_pixbuf_render_to_drawable</code>, sometimes the main event loop). Stepping through the code would cause the problem to disappear (tell-tale signs of a timing bug; those are always the best ones). To make it even more "interesting", sometimes Eclipse would crash altogether with a "BadLength (poly request too large..." error from Xlib. Intuitive remedies, such as upgrading GTK/GDK packages or xorg or xlib, have not been successful, but this idea turned out to be (almost) right.</p>
<p>The breakthrough came after running <code>ldd libswt-pi-gtk-3735.so</code> (this is the SWT library used by Eclipse, which in turn depends on GTK), next determining which package each of the prerequisite libraries shown by ldd belonged to, and successively upgrading each of them. I didn't take care to repeat the test after each upgrade, but here is the list of packages that I replaced or tried to replace - after which the problem disappeared:</p>
<pre>alsa-oss
binutils
gdk-imlib11
libatk1.0-0
libc-dev-bin
libc0.1-dev
libcairo2
libcairo2-dev
libfontconfig1
libfreetype6
libgdk-pixbuf-dev
libgdk-pixbuf-gnome2
libgdk-pixbuf2
libgdk-pixbuf2.0-0
libgdk-pixbuf2.0-dev
libglib2.0-0
libgtk2.0-0
libgtk2.0-common
libnspr4-0d
libpango1.0-0
libpango1.0-common
libpango1.0-dev
libpth20
libpthread-stubs0
libpthread-stubs0-dev
libx11-6
libx1106
libxcb1
libxcomposite1
libxdamage1
libxext6
libxfixes3
libxi6
libxrender1
libxtst6
xorg</pre>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-41074236152022469482011-06-25T17:15:00.004+02:002011-06-25T17:26:43.144+02:00Fix for ClassNotFoundException in PDE JUnit Test Runner<p>When trying to run a PDE JUnit test suite in Eclipse, the requested unit test is not executed, the hosted workbench terminates immediately with the following error to the console (where <code>org.epic.perleditor.editors.TestEditorAssociation</code> is the name of the unit test). The plug-in which contains the allegedly missing class is present in the workspace. (This error also falls into the "it used to work before" category.)</p>
<div style="overflow: scroll; height: 25em; width:100%; white-space: nowrap"><pre>Class not found org.epic.perleditor.editors.TestEditorAssociation
java.lang.ClassNotFoundException: org.epic.perleditor.editors.TestEditorAssociation
at org.eclipse.osgi.framework.internal.core.BundleLoader.findClassInternal(BundleLoader.java:481)
at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:397)
at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:385)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:87)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.eclipse.osgi.framework.internal.core.BundleLoader.loadClass(BundleLoader.java:313)
at org.eclipse.osgi.framework.internal.core.BundleHost.loadClass(BundleHost.java:227)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.loadClass(AbstractBundle.java:1274)
at org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner$BundleClassLoader.findClass(RemotePluginTestRunner.java:38)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClass(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClasses(RemoteTestRunner.java:425)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:445)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner.main(RemotePluginTestRunner.java:62)
at org.eclipse.pde.internal.junit.runtime.UITestApplication$1.run(UITestApplication.java:114)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:133)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3378)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3036)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2382)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2346)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2198)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:493)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:288)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:488)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113)
at org.eclipse.pde.internal.junit.runtime.UITestApplication.start(UITestApplication.java:46)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
at org.eclipse.equinox.launcher.Main.main(Main.java:1212)</pre></div>
<p><strong>Solution:</strong> edit <code>META-INF/MANIFEST.MF</code> of the plug-in containing the not found class and add the class folder (i.e. <code>bin</code>) to the <code>Bundle-ClassPath:</code> directive, like so:</p>
<pre>Bundle-ClassPath: cglib-full-2.0.2.jar,
easymock.jar,
bin/</pre>
<p>If you need to debug this problem in more detail, the method responsible for finding the class (which works in a good configuration, and fails otherwise) is <code>org.eclipse.osgi.framework.internal.core.BundleLoader.findLocalClass</code>.</p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-112080040011486548.post-9710969474708999442011-04-22T13:44:00.011+02:002011-04-22T15:16:04.318+02:00Fix for "Cannot complete the install because of a conflicting dependency."<p>When plug-in installation fails in newer versions of Eclipse (such as "e4"), you may have to deal with cryptic error messages such as this one:</p><div style="overflow: scroll; height: 25em; width:100%; white-space: nowrap"><pre>Cannot complete the install because of a conflicting dependency.
Software being installed: EPIC 0.6.39 (org.epic.feature.main.feature.group 0.6.39)
Software currently installed: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
Only one of the following can be installed at once:
Core Runtime 3.6.100.v20101122 (org.eclipse.core.runtime 3.6.100.v20101122)
Core Runtime 3.7.0.v20110110 (org.eclipse.core.runtime 3.7.0.v20110110)
Only one of the following can be installed at once:
Equinox Launcher Linux X86 Fragment 1.1.100.v20110321 (org.eclipse.equinox.launcher.gtk.linux.x86 1.1.100.v20110321)
Equinox Launcher Linux X86 Fragment 1.1.100.v20101018 (org.eclipse.equinox.launcher.gtk.linux.x86 1.1.100.v20101018)
Only one of the following can be installed at once:
Equinox Java Authentication and Authorization Service (JAAS) 1.1.0.v20110411 (org.eclipse.equinox.security 1.1.0.v20110411)
Equinox Java Authentication and Authorization Service (JAAS) 1.1.0.v20101004 (org.eclipse.equinox.security 1.1.0.v20101004)
Equinox Java Authentication and Authorization Service (JAAS) 1.1.0.v20110124-0830 (org.eclipse.equinox.security 1.1.0.v20110124-0830)
Only one of the following can be installed at once:
Refactoring UI 3.5.100.v20101012-0800 (org.eclipse.ltk.ui.refactoring 3.5.100.v20101012-0800)
Refactoring UI 3.6.0.v20110307-2000a (org.eclipse.ltk.ui.refactoring 3.6.0.v20110307-2000a)
Refactoring UI 3.5.100.v20110111-0800 (org.eclipse.ltk.ui.refactoring 3.5.100.v20110111-0800)
Only one of the following can be installed at once:
Eclipse Platform 4.1.0.v20110420-1420 (org.eclipse.platform 4.1.0.v20110420-1420)
Eclipse Platform 4.1.0.v20110415-1401 (org.eclipse.platform 4.1.0.v20110415-1401)
Eclipse Platform 3.7.0.v201104210100 (org.eclipse.platform 3.7.0.v201104210100)
Eclipse Platform 3.7.0.v201103101119 (org.eclipse.platform 3.7.0.v201103101119)
Eclipse Platform 4.1.0.v20110407-2200 (org.eclipse.platform 4.1.0.v20110407-2200)
Eclipse Platform 4.1.0.v20110415-1030 (org.eclipse.platform 4.1.0.v20110415-1030)
Eclipse Platform 3.7.0.v201103220800 (org.eclipse.platform 3.7.0.v201103220800)
Eclipse Platform 3.7.0.v201104061223 (org.eclipse.platform 3.7.0.v201104061223)
Eclipse Platform 4.1.0.v20110412-2200 (org.eclipse.platform 4.1.0.v20110412-2200)
Eclipse Platform 3.7.0.v201104121532 (org.eclipse.platform 3.7.0.v201104121532)
Eclipse Platform 3.7.0.v201104211800 (org.eclipse.platform 3.7.0.v201104211800)
Eclipse Platform 4.1.0.v20110127-2200 (org.eclipse.platform 4.1.0.v20110127-2200)
Eclipse Platform 3.7.0.v201102150800 (org.eclipse.platform 3.7.0.v201102150800)
Eclipse Platform 4.1.0.v20101202-1530 (org.eclipse.platform 4.1.0.v20101202-1530)
Eclipse Platform 4.1.0.v20110312-1300 (org.eclipse.platform 4.1.0.v20110312-1300)
Eclipse Platform 4.1.0.v20110325-1411 (org.eclipse.platform 4.1.0.v20110325-1411)
Eclipse Platform 4.1.0.v20110419-2200 (org.eclipse.platform 4.1.0.v20110419-2200)
Eclipse Platform 3.7.0.v201012081300 (org.eclipse.platform 3.7.0.v201012081300)
Eclipse Platform 4.1.0.v20110422-0200 (org.eclipse.platform 4.1.0.v20110422-0200)
Eclipse Platform 4.1.0.v20110421-0500 (org.eclipse.platform 4.1.0.v20110421-0500)
Eclipse Platform 4.1.0.v20110414-2200 (org.eclipse.platform 4.1.0.v20110414-2200)
Eclipse Platform 3.7.0.v201104191004 (org.eclipse.platform 3.7.0.v201104191004)
Cannot satisfy dependency:
From: Equinox p2 Provisioning for IDEs 2.1.0.v20110228-897TFncFdHjO2qRuCh_UvYT (org.eclipse.equinox.p2.user.ui.feature.group 2.1.0.v20110228-897TFncFdHjO2qRuCh_UvYT)
To: org.eclipse.equinox.security [1.1.0.v20110124-0830]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.r20110302-9gF7SHCIFt6ms-lIjrC6vK_XO-IabJMKu (org.eclipse.platform.feature.group 3.7.0.r20110302-9gF7SHCIFt6ms-lIjrC6vK_XO-IabJMKu)
To: org.eclipse.platform [3.7.0.v201103101119]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20100923-9fF7MHDqFsAkplGz0n61z-yU57WHoz0JdMfaI (org.eclipse.platform.feature.group 3.7.0.v20100923-9fF7MHDqFsAkplGz0n61z-yU57WHoz0JdMfaI)
To: org.eclipse.rcp.feature.group [3.7.0.v20101115-9FB-FqhFr3P05j0S-HRVgSR]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20110209-9fF7QHFQFsAlpyfO4rKw4lNW200gZcvGr (org.eclipse.platform.feature.group 3.7.0.v20110209-9fF7QHFQFsAlpyfO4rKw4lNW200gZcvGr)
To: org.eclipse.ltk.ui.refactoring [3.5.100.v20110111-0800]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20110315-9gF7SHCJFt3cwDesZ5LrhrcdfooveV1uWqcqPF (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHCJFt3cwDesZ5LrhrcdfooveV1uWqcqPF)
To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyIDVb_FRbwWP]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20110315-9gF7SHIZFt3cwDktE9WrW9XEj1iOiW9fgbhxME (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHIZFt3cwDktE9WrW9XEj1iOiW9fgbhxME)
To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyImVoOcTaz-ZS]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20110315-9gF7SHLgFt4cwDmoWF-DvhVMEUrQfXBVSritR5 (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHLgFt4cwDmoWF-DvhVMEUrQfXBVSritR5)
To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyIxIrTOSarWQ]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20110315-9gF7SHNFFt4cwDooaDrDsaVdEWi0oMLWlh6mbT (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHNFFt4cwDooaDrDsaVdEWi0oMLWlh6mbT)
To: org.eclipse.equinox.security [1.1.0.v20110411]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20110315-9gF7SHNFFt4cwDqmhLPCtURdWWmWmO0Plh6mbT (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHNFFt4cwDqmhLPCtURdWWmWmO0Plh6mbT)
To: org.eclipse.equinox.security [1.1.0.v20110411]
Cannot satisfy dependency:
From: Eclipse Platform 3.7.0.v20110315-9gF7SHNFFt4cwDrmlStE-Ufz0RVqQoc8RS8f2gQ (org.eclipse.platform.feature.group 3.7.0.v20110315-9gF7SHNFFt4cwDrmlStE-Ufz0RVqQoc8RS8f2gQ)
To: org.eclipse.rcp.feature.group [3.7.0.v20110216-9DB5FiuFpBGyY_VrXOSeaUQ]
Cannot satisfy dependency:
From: Eclipse Platform 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU (org.eclipse.platform.feature.group 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU)
To: org.eclipse.ltk.ui.refactoring [3.6.0.v20110307-2000a]
Cannot satisfy dependency:
From: Eclipse Platform 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU (org.eclipse.platform.feature.group 4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU)
To: org.eclipse.platform [4.1.0.v20110312-1300]
Cannot satisfy dependency:
From: Eclipse Product Configuration 1.0.0.I20110310-1119 (org.eclipse.rcp.configuration.feature.group 1.0.0.I20110310-1119)
To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20101018]
Cannot satisfy dependency:
From: Eclipse RCP 3.7.0.v20101115-9FB-FqhFr3P05j0S-HRVgSR (org.eclipse.rcp.feature.group 3.7.0.v20101115-9FB-FqhFr3P05j0S-HRVgSR)
To: org.eclipse.core.runtime [3.6.100.v20101122]
Cannot satisfy dependency:
From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyIDVb_FRbwWP (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyIDVb_FRbwWP)
To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
Cannot satisfy dependency:
From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyImVoOcTaz-ZS (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyImVoOcTaz-ZS)
To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
Cannot satisfy dependency:
From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyIxIrTOSarWQ (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyIxIrTOSarWQ)
To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
Cannot satisfy dependency:
From: Eclipse RCP 3.7.0.v20110216-9DB5FiuFpBGyY_VrXOSeaUQ (org.eclipse.rcp.feature.group 3.7.0.v20110216-9DB5FiuFpBGyY_VrXOSeaUQ)
To: org.eclipse.equinox.launcher.gtk.linux.x86 [1.1.100.v20110321]
Cannot satisfy dependency:
From: Eclipse Project SDK 4.1.0.v20110303-7T7fA7F8Yw_bVbCrFz-gFceimF7IZRyogX1MAdUyRWqYu (org.eclipse.sdk.feature.group 4.1.0.v20110303-7T7fA7F8Yw_bVbCrFz-gFceimF7IZRyogX1MAdUyRWqYu)
To: org.eclipse.platform.feature.group [4.1.0.v20110303-9IF70GdMFnUTMd104-mz-W9mNz0ESwPlAOa-_otA7mYeU]
Cannot satisfy dependency:
From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
To: org.eclipse.equinox.p2.user.ui.feature.group [2.1.0.v20110228-897TFncFdHjO2qRuCh_UvYT]
Cannot satisfy dependency:
From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
To: org.eclipse.rcp.configuration.feature.group [1.0.0.I20110310-1119]
Cannot satisfy dependency:
From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
To: org.eclipse.sdk.feature.group [4.1.0.v20110303-7T7fA7F8Yw_bVbCrFz-gFceimF7IZRyogX1MAdUyRWqYu]
Cannot satisfy dependency:
From: Eclipse SDK 4.1.0.I20110312-1300 (org.eclipse.sdk.ide 4.1.0.I20110312-1300)
To: toolingorg.eclipse.sdk.ide.configuration [4.1.0.I20110312-1300]
Cannot satisfy dependency:
From: EPIC 0.6.39 (org.epic.feature.main.feature.group 0.6.39)
To: org.eclipse.platform.feature.group [3.2.0,4.0.0)
Cannot satisfy dependency:
From: toolinggtk.linux.x86org.eclipse.core.runtime 4.1.0.I20110312-1300
To: bundle org.eclipse.core.runtime 3.7.0.v20110110
Cannot satisfy dependency:
From: toolingorg.eclipse.sdk.ide.configuration 4.1.0.I20110312-1300
To: toolinggtk.linux.x86org.eclipse.core.runtime [4.1.0.I20110312-1300]
</pre></div>
<p>The usability problems with this error message are evident: excessive length, lack of focus, little clue for the developer about possible solutions (and even less for a mere user who is just trying to install a plug-in). Paradoxically, the best way to even know which parts of the error message are relevant at all is to have already solved the problem at hand. Having done just this, here are some possibly helpful inferences:</p><ul>
<li>Chances that you are going to solve a problem of this sort as a normal user are slight. The error likely lies in dependency declarations contained in feature.xml descriptors that are being processed by the installer and you cannot/are not supposed to manually edit those. It's safe to assume that it's the developers' job to sort out this mess. The remaining bullet points address how.</li>
<li>The "Only one of the following can be installed at once" parts of the message are misleading and should be ignored as noise.</li>
<li>The "Cannot satisfy dependency" parts might be relevant, however most of them are bogus, resulting from error propagation, and should be ignored as noise as well.</li>
<li>The root cause is likely contained in the feature.xml descriptor of whatever it is that you are trying to install. In the above example my installation attempt concerns the <a href="http://www.epic-ide.org/">EPIC IDE</a>, and only the following short snippet from the <em>bottom of the error message</em> actually matters:<pre> Cannot satisfy dependency:
From: EPIC 0.6.39 (org.epic.feature.main.feature.group 0.6.39)
To: org.eclipse.platform.feature.group [3.2.0,4.0.0)
</pre></li>
<li>The above dependency comes from a line in content.xml, a file which is found within content.jar downloaded by the installer from the update site. In my case the relevant line of content.xml looked like so:<pre><required namespace='org.eclipse.equinox.p2.iu'
name='org.eclipse.platform.feature.group'
range='[3.2.0,4.0.0)'/>
</pre></li>
<li>The file content.xml is a generated one, so to fix the problem (by removing the line) the developer must track back to the input files that are under his control. In my case it was feature.xml, which contained the following "offending" snippet:<pre> <requires>
<import feature="org.eclipse.platform"
version="3.2.0"
match="compatible"/>
</requires>
</pre></li></ul><p>Eventually, changing match="compatible" to match="greaterOrEqual" in feature.xml and rebuilding content.jar fixed the cause of the original messy error message, as was to be expected ("compatible" means only <a href="http://help.eclipse.org/helios/topic/org.eclipse.platform.doc.isv/reference/misc/feature_manifest.html%3Fresultof%3D%22compatible%22%20%22matching%20rule%22">compatible within the major version</a>, i.e. "3.x.x"). After the fix content.xml contains range="3.2.0", which is fine for installation in 4.1.0.</p>Unknownnoreply@blogger.com12tag:blogger.com,1999:blog-112080040011486548.post-65306764865871045592011-04-05T19:11:00.004+02:002011-04-05T19:26:40.116+02:00Don't use Perl to compute (much)<p>There are times when we want to <a href="http://www.flirble.org/~nick/P/Fast_Enough/">understand and optimize performance of Perl programs</a>, and there are other times when we just want a program to run faster, fast. Having a computation-intensive task at hand (hint: nested loops, numbers, arithmetics) speaks for a brute force approach - simply rewriting the algorithm in C is likely to yield great benefits with little intellectual strain.</p><p>Here's a little case study: a Perl version of a program which performs some 360,000,000 arithmetic operations takes 135 wallclock seconds to execute. After moving the computation to C the execution time drops to 1.7 seconds - a <strong>speedup of more than 70</strong>. Hint: to keep life simple given such a case, don't even bother with perlxs or worry about efficient I/O... just port your code to C, invoke the C program via system() and communicate through files or stdin/stdout.</p><p>(Of course, the same insight certainly applies to any interpreted language, not just Perl.)</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-80666066173586432872011-03-22T18:36:00.007+01:002011-03-23T10:17:40.938+01:00Fix hover display of target links in Firefox 4<p><strong>Problem:</strong> After upgrading to Firefox 4 the first "improvement" that you'll notice is possibly the all-too-distracting popup which displays the full target URL whenever you move the mouse pointer over any link. Apparently they got rid of the status bar, but decided that for security's sake the information must still be displayed somewhere and chose a popup. Unfortunately, a popup which keeps appearing and disappearing over the main page display area feels much more intrusive than a fixed status bar area (which you could also hide altogether).</p><p><strong>Solution:</strong> Install the poorly named add-on <a href="https://addons.mozilla.org/en-us/firefox/addon/status-4-evar/" rel="nofollow">Status-4-Evar</a>. It will instantly restore the original status bar without any further configuration.</p><p>And by the way, you can also disable "View > Toolbars > Tabs on Top" to restore the original proper location of tabs. To move the Home and Reload buttons back to where they belong left of the address entry field, you can right-click on the little arrow within that field (or almost anywhere else on the toolbar), select "Customize" and then drag the buttons back.</p><p>The most comfortable GUIs are the ones we're used to.</p>Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-112080040011486548.post-82371928584583132902011-01-10T20:01:00.004+01:002011-01-10T20:14:37.135+01:00Fullscreen Flash Player on Debian lenny<p><strong>Problem:</strong> Flash plugin crashes in Firefox on 64 bit Debian lenny whenever entering fullscreen mode. The console displays an error message like "libflashplayer.so: undefined symbol: gtk_widget_get_window". Both Firefox and Flash plugin were installed manually rather than from a Debian package on the affected system.</p><p><strong>Solution:</strong> This problem occurs due to an outdated version of 32 bit libgtk2 in Debian lenny (ia32-libs-gtk), which doesn't have the required symbol. As a remedy, you should install 64 bit versions of libgtk2, Firefox and Flash player from Debian packages. However, these packages are not found in lenny repository, so you first need to <a href="http://backports.debian.org/Instructions/">add the backports repository</a> to your apt (important: also add <code>contrib</code> behind <code>main</code> in the repository definition!) After that, just go ahead with<pre>
apt-get -t lenny-backports install flashplugin-nonfree iceweasel libgtk2.0-0
</pre>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-112080040011486548.post-12943397351423364862011-01-06T19:32:00.005+01:002011-01-06T19:40:49.991+01:00Building scalable web applicationsI stumbled upon a presentation by Pressflow founder David Strauss with a nice checklist of common (open source) technologies and buzzwords for building highly scalable web applications. The slides focus on deployment and load/configuration management rather than software design considerations: <a href="http://fourkitchens.com/sites/default/files/PHP%20TEK-X%20-%20Planning%20Infrastructure.pdf">Scalable LAMP infrastructure</a>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-112080040011486548.post-82382339031433250752011-01-05T01:47:00.002+01:002011-01-05T01:58:50.656+01:00Surface drilling Drupal Panels 3<p>This article contains some notes from a short reverse-engineering session targeted at Panels 6.x-3.8 and Chaos Tools 6.x-1.8. The goal was to capture some first impressions and record them for potential future exploration and understanding of that module. The motivation came from the current lack of technical documentation from the powers that be.</p><p>Before deploying the drills, I wish to share with you a few contemplative remarks about Drupal documentation. I feel disappointed by the supposedly "user-friendly" and sometimes overenthusiastic case-oriented explanations so prevalent in descriptions of Drupal ("if you want X, click on Y... how cool!"). Frankly, I find them much inferior to what I like to call model-oriented explanations ("the system consists of parts X, which have state Y and interact according to rules Z") based on strict terminology and aiming to identify intermediate causes at work. A good model capitalizes on your prior software engineering and site building experience by allowing you to generate tons of individual cases in a reasonable amount of time. On the other hand, a set of cases helps you solve only a limited number of problems quickly by imitation. A large database of cases takes much effort to collect, maintain and remember, and it does a poor job at eliminating residual risk. When you're hit with a problem not yet documented in the database and forced to somehow augment your existing cases, this task is going to take an extraordinary amount of time or require to call in heavy artillery (a consultant). In contrast, a well-constructed model takes just a little effort to acquire (but not to create!) and promises much more substantial risk reductions; model refinements do not require quantum leaps of imagination. By abstracting well, you can keep most issues out of your mind most of the time, and with a good choice of metaphors, related pieces of a solution should come together quite effortlessly. In effect, adopting a cohesive model-oriented approach allows you to deliver more consistent problem solving for your clients. Similarly, when you hire someone, they better bring this secret weapon in their arsenal rather than just Google + d.o.</p><p>Back to the main topic. A panel is a named, persistent set of specifications that govern a joint presentation of multiple pieces of content on the same page. These specs cover
</p><ul><li>which kinds of content or even which individual pieces of content should be retrieved from the database;</li>
<li>where (in which pane of a multi-pane layout) each piece of the retrieved content should appear;</li>
<li>how each piece of content should be rendered;</li>
<li>under what circumstances and for whom each piece of content should (dis)appear;</li>
<li>(for panel pages) which URI paths should trigger displaying the specified content.</li> </ul><p>A panel is defined by a site builder. It is stored either in the database (by default) or in code (when exported, e.g. for the sake of <a href="http://en.wikipedia.org/wiki/Revision_control">VCS</a>). It is then used by Drupal as a blueprint for page construction whenever its associated path is requested by the browser. The specifications that comprise a panel are further subdivided into a (non-empty) set of <em>variants</em>, each of which may differ from the other variants in certain respects while sharing the common base configuration to foster reuse. (However, variants are unlike displays from Views in that they all have the same rank; there is no default variant that others would inherit from and override.)</p><p>Panels are somewhat at odds with Drupal core, meaning that they seek to replace rather than just expand upon and integrate with some of the previous ideas:
</p><ul><li>In their capability to <em>lay out</em> pieces of content, panels build upon—but also to an extent compete with—page templates defined by a theme. In particular, panes of a panel layout are conceptually equivalent to regions of a page template. Because a panel is placed
within a region, it can be also thought of as a means for subdividing the region into multiple sub-regions (which are not known to the theme subsystem).</li>
<li>In their capability to <em>retrieve</em> and <em>render</em> pieces of content, Panels developers abandon Drupal's core idea that all content should be represented as either nodes (for user-defined content) or blocks (for reusable pieces of UI). Other kinds of HTML content, not belonging to either category and possibly computed on-the-fly, can also be made available to and rendered in a panel.</li></ul><p>To put the above and following explanations in context, consider the following screenshot of the Panels UI (with some section numbering added):</p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirVKroWwMKve7lP_HDnOgX188AjgZaywLv6T8QE2Vx80a-vl7cPe6Snao38s9sIQALdyaj1Nw37RTEcULeqnq7UEdK-5ST0oeU99__zIrnXJoeILPOTBwcK2lG4gPuBSamGWgAF8vcNlM/s1600/panel.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 222px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirVKroWwMKve7lP_HDnOgX188AjgZaywLv6T8QE2Vx80a-vl7cPe6Snao38s9sIQALdyaj1Nw37RTEcULeqnq7UEdK-5ST0oeU99__zIrnXJoeILPOTBwcK2lG4gPuBSamGWgAF8vcNlM/s400/panel.png" border="0" alt="Panels UI" id="BLOGGER_PHOTO_ID_5558499795021474370" /></a>
<p>The configuration sections are subdivided into three main areas: Summary, Settings, and Variants. The information under Summary and Settings concerns the panel as a whole, while the subsections under Variants pertain to each of the defined named variants. With a few exceptions, the section labels are not self-explanatory at first glance, and the information contained within each section cannot be easily understood separately from the other sections. In this sense,
the UI is non-intuitive. It <em>won't</em> teach what can be done with panels in a learning-by-doing approach. It <em>will</em> cause concern with regard to unexplained settings and overly abstract terminology. The module author's line of defense is that you don't have to understand every piece of the UI to successfully use panels for simple layout scenarios. While true, the suggested adopt-now-explore-later approach also conveys an ominous message: if the documentation is not ever written—and if the adoption or future development of the module suffers—you may find yourself in a bad position by having placed bets on it. (It is not much consolation for new users that the <a href="http://www.lullabot.com/podcasts/podcast-87-panels-vs-context-cage-match">friendly competition</a> doesn't fare much better with regard to making their alternative comprehensible and that the Panels module is well-entrenched with a significant user base.) The question is: can we in the meanwhile remedy this situation by sacrificing some time on the altar of open source and simply peeking into Panels code? Or is the code as poorly done as the documentation? The rest of the article is my first shot at answers (for the impatient: yes and no, respectively). I start by exploring some of the more confusing—while at the same time important—sections of the Panels UI and conclude with a brief foray into code design and implementation issues.</p><p>It turns out that the key to understanding Panels configuration are the concepts of "contexts" and "content". These technical terms are not at all related to the Context module (best erase it out of your mind while studying Panels), only weakly related to the content types of the CCK module, and they might not match any colloquial meaning taken from a dictionary. Such is the nature of terms unleashed on the unsuspecting world at night by well-meaning software developers. Let's focus on "contexts" first. In Panels terminology, the term <em>context</em> refers to a specification of a piece of (renderable) content that should be loaded and become known under a chosen id within the panel variant before it is output as HTML. For example, if your panel should display three nodes next to each other, it would obviously first have to load them, which you would specify by adding three contexts of type node to some panel variant. The Contexts section (2) lets you <em>directly</em> specify the pieces of content that should be loaded; for example specify a node by its nid or title. Alternatively, you could parametrize the panel to load different pieces
of content depending on what has been provided through the URL. To achieve this, you would include placeholders in the panel's path (as shown on the screenshot) and then use the Arguments section (1) to associate each placeholder with a context type (e.g. specify that the first placeholder called <code>%node1</code> really refers to a node ID rather than a taxonomy term ID; at request handling time, the URL is first taken apart by Drupal's menu subsystem and for each placeholder all modules, including Panels, are consulted – see <code>_menu_load_objects</code> in <code>includes/menu.inc</code> for details). Finally, by using the Relationships area of the Contexts section (2) you could also specify a context that loads a piece of content related to another, previously specified, piece of content (say, load the user who is an author of the second node loaded by the node context). Based on the preceding description, a better term for "contexts" might have been "content loaders"; alas, the release night is over, and it is too late for changes now.</p><p>Now let's turn to <em>content</em>. In Panels terminology it refers to "something that can be placed (and somehow rendered in) a pane" through the Content section (3). Unsurprisingly, the pieces of content loaded by contexts as well as their individual parts (such as CCK fields of nodes;
or the user picture from a profile) fall into this category. For this reason you are generally well-advised to first define contexts before even turning to the Content section. However, other pieces of content (such as blocks or HTML snippets) are also available without the need (or possibility) to obtain them through a context. The advantage of loading panel content through contexts is that the attributes of the loaded objects then become available within all panes as variables that can be used in string substitutions (e.g. <code>%node1:title</code> or <code>%user1:name</code>, where the prefix equals context id). For example, having established the three node contexts and one user context as mentioned above, you could add a HTML snippet (aka "custom content") with the node titles and the user name to a pane.</p><p>Having acquired a basic understanding of the context/content pair of concepts and experimented a bit with the Panels UI, here are the next few natural followup questions that come to mind:</p><ul><li>Where do all the selectable context and content types in the Panels UI (and their configuration forms) come from?</li>
<li>How does the Panels module know to offer "Node", "Taxonomy term" and "User" contexts and how do you know what data <em>exactly</em>
they load?</li>
<li>What the heck is a "Node add form" context supposed to load and how is it different from "Node"?</li>
<li>What determines which items appear on the "Miscellaneous" tab when adding content as opposed to the "Widgets"
tab?</li>
<li>What if you want to add something that is <em>not at all</em> listed in Context or Content sections?</li>
<li>How exactly is the Relationships combo populated?</li>
<li>Should we degrade ourselves to click monkeys and figure out all of the above by trial and error?</li></ul><p>I won't even come close to providing detailed (boring?) answers to all of the above questions and will instead address the last (somewhat rhetorical) one by providing a little push in the right direction. As announced before, the solution is sought (and found) by examining the code. It turns out that most of the items and forms you come across while interacting with the Panels UI are not hard-coded anywhere in the Panels module, but rather contributed dynamically by code. "I know – panel hooks!" might be your first guess, but unfortunately it's not that simple. There is a lot more indirection in the Panels code than meets the eye and a more refined mechanism than good old Drupal hooks is at play. Furthermore, it turns out that neither contexts nor content types are designed to be panel-specific. In fact, most of them are not even implemented in the Panels module, but rather belong to the Chaos Tools module, upon which the Panels module depends. In general it's safe to say that the indirection and abstraction sauce is good for flexibility and extensibility and also a definitive poison for code comprehension. The case of Panels is no exception, and so I guess that the author might be playing with fire to some extent ("we have this wonderfully flexible super-duper API for every conceivable task, which unfortunately noone besides us can wrap their head around). I'm glad to report that I have seen much worse and that the code design and inline documentation are definitely more solid than what you see in "advanced help" today. In short, you <em>can</em> wrap your head around it.</p><p>To navigate around Panels code—and to see how the administrative UI is built and what each selectable item stands for—you have to first grasp the much more general concept of CTools Plugins (<a href="http://en.wikipedia.org/wiki/Reification_%28computer_science%29#Reification_in_conceptual_modeling">reified</a> hooks). This part of the Chaos Tools module is an example of the <a href="http://en.wikipedia.org/wiki/Strategy_pattern">Strategy pattern</a> at work. It aims to facilitate a division of labor between module developers and site builders. A plugin instance is an exchangeable persistent object responsible for performing a set of tasks in its own particular way. Plugins are typically selectable (and so instantiated) through the UI of those Drupal modules that depend upon them (like Panels or Views). They may also contain callbacks that construct an administrative form-based UI to allow site builders detailed configuration of each plugin instance. Alternatively, plugins may be organized hierarchically (in a type:subtype or parent:child fashion), for the benefit of a more structured selection UI. The exact set of responsibilities for a plugin type is usually specified by the module developer who invokes plugins of that type from her module. Particular implementations of those responsibilities are then provided as plugins by (potentially) different developers.
A plugin is usually implemented in a single .inc file. Plugin names must be unique within a plugin type, but do not need to be unique across types. Plugins are used by modules or by each other. The CTools library acts as an intermediary and a directory service for publishing, locating
and invoking plugins. This terminology and the discussed relationships are illustrated by the following examples:</p><table border="1" style="border:1px solid black">
<tbody><tr>
<th align="left">Plugin type</th>
<th align="left">Responsibility of plugin type</th>
<th align="left">Plugin</th>
<th align="left">Plugin functionality</th>
<th align="left">Plugin implemented in</th>
<th align="left">Plugin used by</th>
<th align="left">Plugin instance (examples)</th>
</tr>
<tr>
<td>contexts</td>
<td>Provide renderable content for a panel.</td>
<td>node</td>
<td>Load and provide data from a node specified by nid or title.</td>
<td>ctools/plugins/contexts/node.inc</td>
<td>panels.module (indirectly, through its own plugin <i>panels_context</i> of type <i>task_handlers</i>)</td>
<td>node1 and node2 in Variant1 of panelpage1</td>
</tr>
<tr>
<td>"</td>
<td>"</td>
<td>user</td>
<td>Load and provide data from a user profile by uid or name.</td>
<td>ctools/plugins/contexts/user.inc</td>
<td>panels.module (indirectly, through its own plugin <i>panels_context</i> of type <i>task_handlers</i>)</td>
<td>user1 in Variant1 of panelpage1</td>
</tr>
<tr>
<td>content_types</td>
<td>Render available content within an individual panel pane.</td>
<td>node</td>
<td>Render available node content.</td>
<td>ctools/plugins/content_types/node.inc</td>
<td>panels.module (indirectly, through its own plugin <i>standard</i> of type <i>display_renderers</i>)</td>
<td>"Node 1: ID" content in left pane and "Node 2: ID" content in right pane of Variant1 of panelpage1</td>
</tr>
<tr>
<td>"</td>
<td>"</td>
<td>user</td>
<td>Render a picture from user profile.</td>
<td>ctools/plugins/content_types/user_context/user_picture.inc</td>
<td>panels.module (indirectly, through its own plugin <i>standard</i> of type <i>display_renderers</i>)</td>
<td>"Node1 author" user picture in Variant1 of panelpage1</td>
</tr>
</tbody></table>
<p>The required naming conventions and steps for defining plugins (and plugin types) and invoking plugin instances are documented in "advanced help" shipped with the CTools module, which you are advised to read carefully. The CTools library is used pervasively and consistently throughout Earl Miles's (aka <a href="http://drupal.org/user/26979">merlinofchaos</a>) modules (and also in some others). Thus, if we view Drupal as a domain specific language, CTools should count as a potentially prominent dialect. Becoming familiar with it will not only help you understand Panels, but also Views and potentially future of Drupal core, if it should spread into more mainstream use.</p><p style="font-size: 75%;">Note: while looking at the CTools module, I also found that it supports third-party module developers in evolving and versioning their own modules' APIs, including catering to different client modules using different versions of an API at the same time. The term API here refers to the set of hooks which the developed module invokes (i.e. the hooks provided [not the hooks implemented or library functions offered!] by that module). When CTools versioning support is utilized by a module, hook implementors must partition their code into multiple files, each of which corresponds to a particular version of the API. When the hook provider is about to invoke a (particular version of) a hook, it can query CTools for all matching implementations, while excluding any non-matching (e.g. too old or too new) implementations. This mechanism, documented in "e;advanced help", appears to be almost unused, and probably will remain so due to its confusing official description. Nevertheless, it is interesting to see an idea of
this sort in place, given Drupal's track record of neglect for backwards-capability (at least between major versions).</p>Unknownnoreply@blogger.com16tag:blogger.com,1999:blog-112080040011486548.post-89722532323451792962010-12-30T16:27:00.003+01:002010-12-30T17:24:00.440+01:00Drupal multi-sites in a nutshellAlthough Drupal multi-sites are conceptually trivial, I found no satisfactory (short!) explanation in the official documentation or on the web. You can listen to this <a href="http://gotdrupal.com/videos/multisites-vs-multiple-sites">half an hour long presentation</a> like I did, but I'd rather have read the following summary. I suppose you're already familiar with single-site Drupal setups and would like to understand multi-site now:
<ul>
<li>A multi-site configuration consists of a single code base along with multiple databases. Essentially, there is one database per site, although <a href="http://drupal.org/node/291373">relocating some shared tables to a common database</a> (in the same database server) is also possible. This approach should be contrasted with a shared-nothing "multiple site" configuration.</li>
<li>The key to the entire concept is how Drupal locates the <code>settings.php</code> file based on the requested URL. You can steer it to a different <code>settings.php</code> (hence, a different database) for any combination of hostname, port and URI prefix. The exact lookup algorithm and naming conventions are described in <code>settings.php</code> itself.</li>
<li>There is no "master" configuration or "master" site, although the "default" site is used as fallback when a HTTP request matches no specific site (the normal case for non-multi-site setups). Except for that, all sites have the same rank and are unaware of each other. Each site uses the same core code and also the same <code>sites/all/modules</code> and <code>sites/all/themes</code>, while it may also add its own <code>sites/<i>sitedirectory</i>/modules</code> and <code>sites/<i>sitedirectory</i>/themes</code>. If the same module or theme is in present both locations, the site-specific location takes precedence, so you can have diverging versions of modules or themes between sites.</li>
<li>To add another site (or convert your single-site setup to a multi-site setup), simply create a new database for it, add <code>sites/<i>sitedirectory</i></code>, copy <code>sites/<i>sitedirectory</i>/default.settings.php</code> to <code>sites/<i>sitedirectory</i>/settings.php</code> and visit the new site's URL in the browser. The usual Drupal installation procedure will be offered.</li>
<li>The <code>files</code> folder is usually placed under the site-specific folder, i.e. separate for each site (if you want it shared, you should also take care to share the <code>files</code> database table and possibly others).</li>
<li>Because the databases are separate, each site can have a different subset of all the available modules/themes enabled.</li>
<li>Because the databases are separate, you have to run update.php separately for each site when upgrading Drupal core or contributed modules.</li>
<li>Because the Drupal core is shared, you may run into dependency problems if at least one site requires a different version of Drupal core than the others.</li>
<li>Because the web server user and file permissions are shared, the multi-site configuration is <em>potentially insecure</em>: any site administrator who can run PHP code may access files (and, by peeking into settings.php, also gain full access to the database) of any site.</li>
</ul>Unknownnoreply@blogger.com0