Debugging serialization in SOAP::Lite

Scenario: you are using the Perl SOAP::Lite module as a SOAP client, and want to invoke a web service operation which accepts a complex data type. You provide SOAP::Lite with an appropriately structured SOAP::Data object that represents the value of your type. However, on the server side you notice that xsi:nil is transmitted instead of the value.

The solution is to define a seralizer method for your complex type, like so:

sub SOAP::Serializer::as_name_of_your_complex_type
{
    my $self = shift;
    my $value = shift;
    my $name = shift;
    my $type = shift;
    my $attr = shift;

    return [$name, { 'xsi:type' => 'tns:name_of_your_complex_type', %$attr }, $value];
}

The rest of this post elaborates how one would arrive at this solution and provides a short study of SOAP::Lite internals.

The perils of understanding SOAP::Lite

Debugging SOAP::Lite is not easy. In fact, I bet that the author of SOAP::Lite does not use a debugger himself. Apparently, he didn't spend a lot of time pondering the cruelty to his readers inflicted by a whole arsenal of Perl tricks. Let me enumerate several issues:

1. The code is long. This reduces your chances of finding a portion relevant to your problem by the sheer luck of skimming over it.
2. It heavily relies on Perl automatic (predefined) variables. Your memory will be severely strained if you try to read this code. The author possibly assumes that Perl is your only language and mother tongue, too.
3. It uses generated code and late initialization to great extent. Of course, you can't set breakpoints in the generated code. It's also rather difficult to SEE the generated code in all its glory, for this requires you to first establish the numerous points of its generation. You will also get dizzy from the frequent back-and-forth transfers of control between subroutines that implement actual functionality and generic "internal plumbing" subroutines that have little to do with the main thread of execution. Forget stepping through SOAP::Lite code. At best, you can stumble through it.
4. It uses nested expressions a lot - long live functional programming! Too bad that it defeats the idea of setting line breakpoints.
5. It follows the "self-documenting genius code" style. Natural language explanations of the code are unnecessary. Even worse, it doesn't contain any specifications (parameter/return value/contracts) for the subroutines. You have to infer each subroutine's intent and valid invocation scenarios solely based on its implementation. Surely, there is a method to this madness, but it's up to you to find it.

All that said, one must admit that SOAP::Lite has been around for a very long time, works very well (until it ceases working, that is), it is the only such comprehensive SOAP module around, and it is of course free. These factors mean that we have to adjust to its author's preferred programming style rather than vice versa (and also thank him for sparing us the effort of implementing the SOAP protocol ourselves).

A walkthrough toward generated code

There are plenty of ways to interact with SOAP::Lite. For the following descriptions, assume that you use something like that:

my $service = SOAP::Lite        
        ->service("$url/wsdl/inventory.wsdl")
        ->proxy("$url/cgi-bin/soap_server.pl");

my $result = $service->dispatch_to(
        $test_user,
        $test_password,
        'inventory_update',
        $args
        );

...where $test_user, $test_password are strings and $args is a SOAP::Data object containing the value of a complex type.

You begin your interaction with SOAP::Lite by invoking the static service method. This method is a factory method - it invokes the SOAP::Lite constructor to create a new SOAP::Lite object. You don't get a SOAP::Lite instance from it, though. Rather, the object is blessed into a (generated) subclass whose name is based on the name attribute of the wsdl:service element in the WSDL you specified as parameter. In our example case, the subclass is called 'dispatch_to_service'.

Two noteworthy attributes of the newly created object are _serializer and _deserializer, of type SOAP::Serializer and SOAP::Deserializer, respectively. Other attributes' names also start with an underscore.

The created service object at first appears to have no methods of its own. Various methods are added to this object at various points of execution, though. For example, when you attempt to invoke the method proxy, the code generator kicks in. In this case, a subroutine (?) named SOAP::Lite::BEGIN generates non-underscore-named accessor (get/set) methods for all (most of?) the underscore-named attributes. Also generated is the 'proxy' method, which delegates to the the same-named method on the SOAP::Transport object stored in the _transport attribute. This one in turn creates a SOAP::Transport::HTTP::Client object, which is stored in the _proxy attribute of the service object and henceforth returned whenever you call $service->proxy.

Within your invocation of SOAP::Lite::service, the generation of stubs for wsdl:operations of the service (that is, the actual business logic methods) also occurs. More precisely, the following line from Lite.pm is responsible for invoking the generator:

my %services = %{$self->schema->parse(@_)->load->services};

The subroutine SOAP::Schema::load iterates over all the operation names and invokes generate_stub for each operation. The generated stub code is stored in the _stub attribute of a SOAP::Schema object. Actually, the operations are not even placed as normal methods in the generated stub code, but rather (again) generated using the AUTOLOAD mechanism by the stub code. Here is an example of what it looks like in all its generated glory. Actually getting to the point of seeing this code took me a few hours debugging:

package dispatch_to_service;
# Generated by SOAP::Lite (v0.710.08) for Perl -- soaplite.com
# Copyright (C) 2000-2006 Paul Kulchenko, Byrne Reese
# -- generated at [Thu Mar  5 14:05:39 2009]
# -- generated from http://localhost/wsdl/inventory.wsdl
my %methods = (
dispatch_to => {
    endpoint => 'http://localhost/cgi-bin/soap_server.pl',
    soapaction => 'http://www.plosquare.com/wsdl/WebServices#dispatch_to',
    namespace => 'http://www.plosquare.com/wsdl/WebServices',
    parameters => [
      SOAP::Data->new(name => 'user', type => 'xsd:string', attr => {}),
      SOAP::Data->new(name => 'password', type => 'xsd:string', attr => {}),
      SOAP::Data->new(name => 'function', type => 'xsd:string', attr => {}),
      SOAP::Data->new(name => 'args', type => 'tns:inventory_argument_type', attr => {}),
    ], # end parameters
  }, # end dispatch_to
); # end my %methods

use SOAP::Lite;
use Exporter;
use Carp ();

use vars qw(@ISA $AUTOLOAD @EXPORT_OK %EXPORT_TAGS);
@ISA = qw(Exporter SOAP::Lite);
@EXPORT_OK = (keys %methods);
%EXPORT_TAGS = ('all' => [@EXPORT_OK]);

sub _call {
    my ($self, $method) = (shift, shift);
    my $name = UNIVERSAL::isa($method => 'SOAP::Data') ? $method->name : $method;
    my %method = %{$methods{$name}};
    $self->proxy($method{endpoint} || Carp::croak "No server address (proxy) specified")
        unless $self->proxy;
    my @templates = @{$method{parameters}};
    my @parameters = ();
    foreach my $param (@_) {
        if (@templates) {
            my $template = shift @templates;
            my ($prefix,$typename) = SOAP::Utils::splitqname($template->type);
            my $method = 'as_'.$typename;
            # TODO - if can('as_'.$typename) {...}
            my $result = $self->serializer->$method($param, $template->name, $template->type, $template->attr);
            push(@parameters, $template->value($result->[2]));
        }
        else {
            push(@parameters, $param);
        }
    }
    $self->endpoint($method{endpoint})
       ->ns($method{namespace})
       ->on_action(sub{qq!"$method{soapaction}"!});
  $self->serializer->register_ns("http://schemas.xmlsoap.org/wsdl/","wsdl");
  $self->serializer->register_ns("http://www.w3.org/2001/XMLSchema-instance","xsi");
  $self->serializer->register_ns("http://schemas.xmlsoap.org/soap/encoding/","soapenc");
  $self->serializer->register_ns("http://www.w3.org/2001/XMLSchema","xsd");
  $self->serializer->register_ns("http://www.plosquare.com/wsdl/WebServices","tns");
  $self->serializer->register_ns("http://schemas.xmlsoap.org/wsdl/soap/","soapbind");
    my $som = $self->SUPER::call($method => @parameters);
    if ($self->want_som) {
        return $som;
    }
    UNIVERSAL::isa($som => 'SOAP::SOM') ? wantarray ? $som->paramsall : $som->result : $som;
}

sub BEGIN {
    no strict 'refs';
    for my $method (qw(want_som)) {
        my $field = '_' . $method;
        *$method = sub {
            my $self = shift->new;
            @_ ? ($self->{$field} = shift, return $self) : return $self->{$field};
        }
    }
}
no strict 'refs';
for my $method (@EXPORT_OK) {
    my %method = %{$methods{$method}};
    *$method = sub {
        my $self = UNIVERSAL::isa($_[0] => __PACKAGE__)
            ? ref $_[0]
                ? shift # OBJECT
                # CLASS, either get self or create new and assign to self
                : (shift->self || __PACKAGE__->self(__PACKAGE__->new))
            # function call, either get self or create new and assign to self
            : (__PACKAGE__->self || __PACKAGE__->self(__PACKAGE__->new));
        $self->_call($method, @_);
    }
}

sub AUTOLOAD {
    my $method = substr($AUTOLOAD, rindex($AUTOLOAD, '::') + 2);
    return if $method eq 'DESTROY' || $method eq 'want_som';
    die "Unrecognized method '$method'. List of available method(s): @EXPORT_OK\n";
}

1;

Accordingly, if you want to debug your stubs, you should incorporate your debugging output in the generator routine SOAP::Schema::generate_stub. Back in the beginning of the post, the problem was that a complex data structure was being serialized into xsi:nil rather than into the provided value. Inserting debugging output would have shown that the line

my $result = $self->serializer->$method($param, $template->name, $template->type, $template->attr);

in the generated stub doesn't yield the expected value for the 'args' parameter and would provide a clue to the custom serialization method as a method of solving the problem.

Hopefully you enjoyed this little trip into SOAP::Lite internals as much as I did making it on my own (that is to say, just a bit ;-). Seriously, this post serves to capture several facts about SOAP::Lite that might aid debugging if (when) the next such necessity arrives.