Classloading from Google App Engine's data store

I have been playing around with the ideas of mobile processing agents lately, and to do this I need to execute code that has been uploaded to my processing environment, rather than the traditional “deployed” code. Java has this ability built in, via ClassLoaders. At any point, a java application can create classes from byte arrays. I wanted to do this on Google App Engine, but GAE does not support the traditional methods of loading classes would work. There are no “Files” or direct access to URLs in GAE. The only real data store available is the Data Store, which provides a HTTP based file upload and storage service through its own proprietary interface.

Creating a ClassLoader which took advantage of the Data Store wasn’t difficult however. I made use of the (currently experimental) File Store API to access the code once it has been uploaded to GAE’s datastore using the file upload facility. Once I’d gotten that sorted out, it was simply a matter of writing the classloader to read in the data.

To make things a bit more interesting, I’ve also added an AppLoader, which can set up a classloader for a contained application with extra JARs and a manifest, similar in structure to a WAR file. For example, if we uploaded a JAR file to GAE with the following structure

META-INF/agent-inf.yaml
classes/com/blah/blah/Foo.class
classes/com/blah/blah/blah/Bar.class
classes/com/blah/blah/blah/Etc.class
lib/addon.jar
lib/another.jar

The AppLoader would construct a classloader with the contents of the classes folder, plus any JAR files that are in the lib dir. The agent-inf.yaml file specifies which class is the “main” class for the agent, which will be used for execution.

There is one limitation to the classloader that I’ve written. Sometimes java code refers to non-class files that are stored on the classpath, usually referred to as Resources. They are loaded by the Class.getResource() and Class.getResourceAsStream() methods. getResource() returns a URL which points to the resource asked for. The problem that I have is that GAE does not support URLs properly, and certainly doesn’t allow you to have your own URL handlers. As a result, getResource() does not work in my classLoader. This may break some libraries. getResourceAsStream() does work however.

One final warning about isolation and Classloaders. Giving people access to upload arbitary code to your app server is a dangerous undertaking. Make sure you understand who is uploading classes and make sure they can’t do anything they shouldn’t be doing. In my agent code, I want each of the agents to be completely isolated from each other. The challenge that I have is that each one, if it has access to the DataStore API, will be able to overwrite the others, which isn’t cool. I’m still thinking about ways that I can fix that.

In case anyone else is looking for a similar solution, I thought I’d post it here. The Code for this example is available at my GitHub Repo

Now, back to writing that mobile processing agent system…