Solution for GSS-API major_status:00090000, minor_status:861b6d0c

Problem: You are trying to configure mod_auth_kerb to work with Active Directory. You have created the technical user account "yourserver" for HTTP service in AD and associated it with the Kerberos SPN HTTP/your.server.com@YOURDOMAIN.COM using ktpass.exe. When accessing the page which requires Kerberos auth with IE, you see the following error messages in Apache error log (after raising LogLevel to debug):
[client nnn.nnn.nnn.nnn] Warning: received token seems to be NTLM, which isn't supported by the Kerberos module. Check your IE configuration.
src/mod_auth_kerb.c(1101): [client nnn.nnn.nnn.nnn] GSS-API major_status:00090000, minor_status:861b6d0c
[client nnn.nnn.nnn.nnn] gss_accept_sec_context() failed: A token was invalid (, Unknown code)
You have checked your IE (or Firefox) configuration and are pretty sure that the browser should be sending a Kerberos ticket instead of attempting NTLM authentication.

Solution: First apply some more diagnostics:
  1. Check if the command
    kvno HTTP/your.server.com@YOURDOMAIN.COM
    gives you this message: "HTTP/your.server.com@YOURDOMAIN.COM: Server not found in Kerberos database while getting credentials". If yes, you likely have the problem described here.
  2. On the Windows AD server check the output of setspn -l yourserver. Does it appear like so?
    Registered ServicePrincipalNames for CN=yoursever,OU=Service Accounts,OU=Accounts,DC=yourdomain,DC=com:
            HTTP/your.server.com@YOURDOMAIN.COM
    If yes, then you likely have the problem described here. The domain suffix highlighted in red should not appear in the command's output in a correct configuration.
To fix the incorrect SPN association, use the following commands on AD server:
setspn -d HTTP/your.server.com@YOURDOMAIN.COM yourserver
setspn -A HTTP/your.server.com yourserver

Solution for GSS-API major_status:000d0000, minor_status:96c73ae6

Problem: You are trying to configure mod_auth_kerb. When you access a page that requires Kerberos auth with IE, a popup asking for password appears. In Apache error log (after raising LogLevel to debug), you can see the following messages appearing:
[debug] src/mod_auth_kerb.c(1101): [client nnn.nnn.nnn.nnn] GSS-API major_status:000d0000, minor_status:96c73ae6
[error] [client nnn.nnn.nnn.nnn] gss_accept_sec_context() failed: Miscellaneous failure (, Unknown code)
Solution: Check that the kvno (key version number) stored in your keytab matches the kvno reported by the kvno command. You can view kvno from keytab using
klist -Kekt /path/to/keytab
Compare this against
kvno HTTP/your.server.com
The kvno command only works after a successful authentication with kinit. You can recreate your keytab with the expected kvno (in the example below: 3) using ktutil:
add_entry -password -p HTTP/your.server.com@YOURDOMAIN.COM -k 3 -e RC4-HMAC
wkt /path/to/newkeytab

Solution for "Key table entry not found while getting initial credentials"

Problem (with Kerberos on Debian stable):
kinit -V -k -t /tmp/keytab HTTP/somehost@LOCALDOMAIN
kinit: Key table entry not found while getting initial credentials
However klist -k -t /tmp/keytab works and correctly displays HTTP/somehost@LOCALDOMAIN as present under Principals. The keytab entry was previously created with using addent -password -p HTTP/somehost@LOCALDOMAIN -k 1 -e rc4-hmac.
Solution:
Had to adjust /etc/krb5.conf:
default_tkt_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5
default_tgs_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5
Source link

Running OQL queries from command line

OQL is a specialized query language for memory profiling of Java heap dumps. It is implemented by tools such as jvisualvm and jhat, which ship with current versions of JDK. Unfortunately, neither of these tools provides a simple command-line interface out of the box to integrate them into your own development environment and support real-life scripting. Rather, each of them gives you a kludgy and insufficient GUI.

However, it is easy simply to feed queries to the web server provided by jhat. You can download the OQL command line Perl script to do just that (see documentation).

A few notes from my playing around with OQL:

  • The ability to define your own Javascript functions is very useful, go for it!
  • Unfortunately, some of the handy built-in functions provided by jvisualvm are absent from jhat (need to define your own).
  • To obtain object IDs that can be passed to heap.findObject you must use the built-in function objectid(objectReference). The identifiers displayed by jvisualvm that look like #nnn are not objectids (that would be too easy, eh? I don't see any way to obtain them using jvisualvm GUI, have to use OQL).
  • To display the entire content of a char[] array (which jvisualvm cuts off in a useless way), you must use an OQL expression like objectReference.toString().
  • If OQL scares you, consider buying a commercial profiler.

Software as a process

I have always had a (largely concealed) aversion to the term "software architecture". Indeed I even used to think: the less architecture you had, the better. "Software architecture" seemed to me as an euphemism for "premature constraints" that limited the directions into which your software might evolve; the new clothes on the widely disregarded "waterfall" model. Furthermore, before the rather pompous sounding ideas of "software architecture" were invented (and then standardized by ISO commitees), we had already had notions of good "software design" covering about the same themes.

I was also suspicious about comparing software to other disciplines such as civil engineering where physical structures are erected. Another analogy, city planning, commonly invoked by those colleagues who prefer to call themselves "enterprise architects" was somewhat less alien, but still not quite appealing for some reason.

I used to attribute that apprehension to my perceived malleability of software, the relative ease of changing it, adding to it, reconfiguring it. I still think that is a valid reason to object to analogies that also let us think about software lifecycle as "build (or buy)", "use", "replace" phases and that inescapably guide our thoughts toward the established notion of licensed "software products". But I feel there is also a nimbler mental notion which neatly explains my discomfort with those mainstream ideas: software as a process.

Processes - as in "business processes" or "industrial processes" - are a way of modelling the world in terms of events, changes and dependencies. Good things happen. Bad things happen. Resources are consumed and produced. Information starts and stops flowing between entities. Entities come into existence. Entitites disappear. Databases grow. Users request services. Customers are won and lost. Cash flows. The world is in motion. Careful, continuous process control at all levels is what we need to avoid costly mistakes.

Our role as software developers is often not to invent some sort of "best practice architecture" and then naively stick to it, not even to bring a specified product to market. Instead, it is to react to environmental changes and guide our software through a path of constant change. It is to ensure that we maintain control of that what we once built - and keep extending, adapting and improving - and to design our software in such a way that the broader processes in which it participates are supported rather than hindered. It is to anticipate scenarios with risks and opportunities and to make them less or more probable to fulfill business objectives. Such scenarios always concern both the currnet use and the future evolution of the systems we create.

By the way, the distinction between "programming in the large" and "programming in the small" is FUD. When viewed through the lenses of process control, both areas indeed appear much the same, unified even. The structure of our engineering problems is more akin to a fractal or a piece of music: the same (few) recurring themes and prinicipal rules of composition appear both in the large and in the small. However, few people flip-flop between work at different scales, which leads to an unfortunate multiplication of different names for similar concepts.

Today I am relieved to see that for several years I haven't been the dumb one who was stubbornly rejecting the grand concepts of software architecture and marketing myself as some awkward combination of "developer" and "systems administrator" (a jack of all trades...) at my own peril. The ideas outlined above now seem to be gaining increasing popularity among highly respected practitioners charged with maintaining big software systems in big (Internet) companies. Lo and behold, there even seems to be a catchy buzzword: "devops", a mix of the traditional roles "developement" and "operations".

By the way, I am not opposed to the job role of "software architect", if you interpret it as someone who is hired by a client to keep watch against imagination of inexperienced software developers (although "software process manager", "master devop" or simply "chief programmer/designer" might still be more accurate descriptions of the job).

Last, but not least: YMMV. The world of software development is diverse that it is likely that no single mental model will fit all imaginable contexts; or that the signle grand unified model may prove too abstract to be immediately useful. It makes quite a practical difference if you are designing shrink-wrapped software products for consumers or planning or maintaining installations of enterprise or Internet SaaS applications. Pick your specific models wisely, so as to benefit your own work.

Uncorrelated subqueries treated as DEPENDENT by MySQL

Are you using the operator IN with subqueries in MySQL and wondering about poor performance? Have you tried EXPLAINing your queries and noticed that they are reported as "DEPENDENT SUBQUERY" despite your best intent to use an uncorrelated subquery? If so, read on:

This post describes a MySQL bug (some call it a feature) which has been open for many years and according to official documentation still exists in version 5.6 (note that I only checked 5.0.x and 5.1.x myself and didn't bother to look for info about 6.0, but I have no high hopes based on the bug's age and character).

Consider the following two SQL statements:

  1. SELECT id FROM t1 WHERE id IN (1, 2, 3)
  2. SELECT id FROM t1 WHERE id IN (SELECT id FROM t2)

In MySQL they have very different performance characteristics: the first one will work fast as expected, the second one will actually cause index lookups to be executed repeatedly for each row from t1 rather than just once for the entire statement. Why?

Blame the "optimizer", as explained in the manual. Our fine uncorrelated query becomes forcingly "optimized" into an EXISTS query and thus correlated (rougly speaking; it's a bit more complicated, but you can view the result by using EXPLAIN EXTENDED followed by SHOW WARNINGS). MySQL of course must handle correlated queries per row, leading to disastrous algorithmic performance in what one would assume is a very common case.

I have no clever workaround - you can't "convince" the MySQL optimizer to behave differently. What you can do is use temporary tables to simulate what you really want to happen. But if you do, beware that you may only refer to a temporary table once in a SELECT statement due to another hugely annoying limitation.

For further information see:

P.S. If you found a MySQL version that works in a smarter way, don't hesitate to drop a comment.

Adding attributes to SOAP-Lite envelope

There is a common theme in developing software while having to use badly designed libraries: you know exactly what you want to achieve, and are not asking for much, but a particular library prevents you from progressing. Without any doubt, SOAP::Lite in Perl is this kind of a library. Suppose we want to perform some simple modification of the generated XML SOAP message, such as adding another attribute to the Envelope element. This shouldn't be hard at all given the purpose of the library. Yet it turns out otherwise. Is there an officially blessed way of doing it with SOAP::Lite? I don't know. But there is this hack:

    $envelope_orig = \&SOAP::Serializer::envelope;
    *SOAP::Serializer::envelope = sub {
        my $str = $envelope_orig->(@_);
        $str =~ s{xmlns:xsi=}{xsi:schemaLocation="http://whatever" xmlns:xsi=}s;
        return $str;
    }

By replacing the SOAP::Serialize::envelope subroutine you can alter the resulting XML message in whatever way you wish. This is working around the API, a potentially dangerous thing to do in general (though in this particular case, the danger of SOAP::Serialize::envelope being removed in a next version is slim). Use at your own risk.