Migrating Eclipse update sites to P2

This article describes how to upgrade an Eclipse update site from the classic, pre-P2 layout (site.xml together with plugins and features directories) to the "new and improved" (read: unnecessarily complicated) P2 layout. Considering that P2, the new update manager introduced in Eclipse 3.4, is backwards-compatible with the old layout of update sites (more or less), why bother at all? In my experience, browsing update sites with many plug-ins and features used to be somewhat slow before, but with P2 it has become excruciatingly slow. Fortunately, P2 also supports a new update site format designed to alleviate this problem, that is, to speed up update site browsing. However, as far as I know, no comprehensive documentation of the upgrade path for those of us still using the classic update site layout exists. More confusingly, the article Update Site Optimization (coming from the source?) only tells half of the story. As I found out, following the somewhat outdated instructions listed in there won't leave you with a working update site. Information from different sources had to be pieced together. Hopefully, no longer, if you read on.

Classic update site vs. P2 update site

Let's start by comparing the classic update site layout with the newer layout tailored for P2. I suppose you have a working update site organized in the classic way and want to switch to the new P2 one for reasons mentioned above:

Classic update site	P2 update site
update site = a world-accessible directory on a web server	same as before
update site contains site.xml, which contains a list of versioned features installable from this site	update site contains content.jar and artifacts.jar, which together supersede site.xml
subdirectory features contains one JAR file per versioned feature, referenced from site.xml	same as before, but the references now originate from content.jar and artifacts.jar
subdirectory plugins contains one JAR file per versioned plug-in, referenced from feature.xml files contained in feature JARs	subdirectory plugins contains one .jar.pack.gz file per versioned plug-in, referenced from feature.xml as before, but also from content.jar and artifacts.jar
	update site (optionally?) contains digest.zip - see description below

While the syntax of feature.xml and site.xml is easy to understand and these files are easy to generate/process by your own tools if need be, the new content.jar and artifacts.jar leave no such hopes. You are advised to treat them as P2's private mess (found inside: fat, obscure XML documents). As you will see next, both these JAR files can and should be generated from a slightly modified version of site.xml.

The plug-in .jar.pack.gz files are also generated - each from the original plug-in JAR found of the classic update site. They are essentially JARs recursively compressed with the pack200 tool first introduced in Java 1.5.

The old site.xml and old plug-in JAR files are not ever accessed by the P2 update manager, but if you wish to stay backwards-compatible, you should keep them around in your update site as well.

If you read the Update Site Optimization article, you might be wondering whether a "digest" file (digest.zip) is also needed, where to put it, and what for. I have never observed P2 trying to access this file and suspect that it has been superseded by the content.jar and artifacts.jar duo (which the original article fails to mention). However, it is worth noting that the official Eclipse Ganymede update site does contain a digest.zip in the update site directory and an attribute digestURL="http://download.eclipse.org/releases/ganymede/" on the site element in site.xml (the value of this attribute points to the update site, i.e. location of digest.zip). We'll see next how to generate digest.zip just in case.

How to upgrade to a P2 update site

In essence, the following steps are required:

1. Add a new attribute pack200="true" to the site element in site.xml.
2. Add a new attribute digestURL="http://your/update/site/url/" to the site element in site.xml.
3. Ensure that each of your feature JARs in the features directory contains a feature.properties file (which may be empty). (Here is why.)
4. Generate a .jar.pack.gz file from each plug-in JAR file in the plugins subdirectory.
5. Generate digest.zip based on the classic update site (including site.xml).
6. Generate content.jar and artifacts.jar based on the classic update site (including site.xml).

Steps 1, 2, 3 do not require any sophistication and thus won't be further elaborated. Steps 4 and 5 are (surprisingly) intertwined, as described below. Step 6 is also described in the last section.

How to generate .jar.pack.gz files (Step 4) and digest.zip (Step 5)

The generating .jar.pack.gz files step actually consists of two parts:

1. For each JAR file in question, "condition" or "repack" the JAR file to prepare it for the second part.
2. Generate .jar.pack.gz files from a set of conditioned JAR files. Also generate digest.zip

In order to condition (repack) a JAR file, run:

$JAVA_HOME/bin/java \
    -jar $launcher \
    -application org.eclipse.update.core.siteOptimizer \
    -jarProcessor -verbose -processAll -repack -outputDir $output_dir/plugins \
    $input_jar_file

The above invocation contains some variables, to be replaced as follows:

JAVA_HOME	path to where Java 1.5 (or newer) is installed
launcher	path to plugins/org.eclipse.equinox.launcher_1.0.101.R34x_v20080819.jar from your Eclipse 3.4 (or newer) installation (the version number in the JAR file name may vary)
output_dir	path to where the conditioned JAR file should be written - the plugins directory of the upgraded update site
input_file	path to the input plug-in JAR file

Sadly, you will have to run the above tool for each JAR file individually. Observe that the conditioned JARs contain META-INF/eclipse.inf, absent in unconditioned ones. Finally, copy site.xml and the features subdirectory of your original update site to $output_dir. This completes part 1.

For part 2, run the following command. Unlike part 1, this processes all the conditioned JARs it can find through site.xml:

$JAVA_HOME/bin/java \
    -jar $launcher \
    -application org.eclipse.update.core.siteOptimizer \
    -digestBuilder \
    -digestOutputDir=$output_dir \
    -siteXML=$output_dir/site.xml \\
    -jarProcessor -pack -outputDir $output_dir $output_dir

You should now have $output_dir/plugins full of conditioned JARs and a corresponding .pack.jar.gz for each of them. You should also have $output_dir/digest.zip. If not, maybe you forgot to take care of feature.properties in Step 3 mentioned earlier.

How to generate content.jar and artifacts.jar (Step 6)

Here is how (set a human readable $project_name first):

$JAVA_HOME/bin/java -jar $launcher \
    -application org.eclipse.equinox.p2.metadata.generator.EclipseGenerator \
    -updateSite ${output_dir}/ \
    -site file:${output_dir}/site.xml \
    -metadataRepository file:${output_dir}/ \
    -metadataRepositoryName "${project_name} Update Site" \
    -artifactRepository file:${output_dir}/ \
    -artifactRepositoryName "${project_name} Artifacts" \
    -compress \
    -reusePack200Files \
    -noDefaultIUs \
    -vmargs -Xmx256M

The final result

Here is an example P2-upgraded, backwards compatible update site with 1 feature version:

.
|-- site.xml
|-- artifacts.jar
|-- content.jar
|-- digest.zip
|-- features
|   `-- org.epic.feature.main_0.6.34.jar
`-- plugins
    |-- org.epic.debug_0.6.27.jar
    |-- org.epic.debug_0.6.27.jar.pack.gz
    |-- org.epic.doc_0.6.2.jar
    |-- org.epic.doc_0.6.2.jar.pack.gz
    |-- org.epic.lib_0.6.1.jar
    |-- org.epic.lib_0.6.1.jar.pack.gz
    |-- org.epic.perleditor_0.6.23.jar
    |-- org.epic.perleditor_0.6.23.jar.pack.gz
    |-- org.epic.regexp_0.6.1.jar
    |-- org.epic.regexp_0.6.1.jar.pack.gz
    |-- org.epic.source_0.6.34.jar
    `-- org.epic.source_0.6.34.jar.pack.gz

site.xml contains the two new attributes, pack200 and digestURL.

P2 will access content.jar, artifacts.jar, org.epic.feature.main_0.6.34.jar and the plug-in .jar.pack.gz files.

The classic update manager will access site.xml, org.epic.feature.main_0.6.34.jar and the plug-in .jar files.

Automation of the above process using Ant scripts, predefined Eclipse tasks or some such is left as an exercise for the reader. Also note another post with hints on how to debug helper applications started by the Eclipse launcher.