Updated 2009-06-22: Added settings that limit JetS3t to a single HTTP connection at a time, to work around apparent thread-safety issues in Walrus.
Eucalyptus is a relatively new but rapidly developing open-source system for running your own cloud computing clusters. There have been some exciting announcements recently around this project: it has been added to Ubuntu 9.04 as the Ubuntu Enterprise Cloud technology preview, integrated into RightScale's cloud management service, and has received some venture funding.
What does all of this mean? It is becoming easier to create your own cloud computing platform using open-source tools, and then to combine your own cloud with public services like Amazon's AWS in interesting ways.
As well as helping you manage your own cloud computing clusters,
Eucalyptus includes a storage service called Walrus that is (almost)
interface compatible with Amazon's own S3 storage service. Great! So now
you can have your own personal S3-style storage service, but surely
there should be an easier way to interact with this storage than using
the s3curl.pl tool as described in the Walrus documentation?
Happily, the upcoming version of JetS3t will be compatible with the Walrus. Goo goo ga joob.
Configuring JetS3t for Eucalyptus/Walrus
To use JetS3t with the Eucalyptus storage service you will need version 0.7.1 -- this probably won't be available on the download page until May 8th or so, but you can grab the pre-release code from the CVS repository and build it yourself if you are keen (a java.net account is required).
Once you have the latest version, you will need to configure JetS3t to
talk to your own Walrus service by editing the jets3t.properties file
in the configs directory. In summary, you must tell JetS3t where your
Eucalyptus cluster service is located and turn off the HTTPS and
DNS-naming features that are not yet supported by Walrus.
Here are the non-default settings I used to talk to a Eucalyptus front-end running on my own server with the IP address 192.168.1.128:
s3service.s3-endpoint=192.168.1.128
s3service.s3-endpoint-http-port=8773
s3service.s3-endpoint-virtual-path=/services/Walrus
s3service.https-only=false
s3service.disable-dns-buckets=true
Depending on the version of Walrus you are using, you may need to apply some additional settings. As of June 2009, there is some evidence that Walrus is not entirely thread-safe and will behave badly if you perform multiple simultaneous requests. You can work around this issue while still taking advantage of JetS3t's multi-threaded tools by configuring JetS3t to use only one HTTP connection at a time:
s3service.max-thread-count=1
s3service.admin-max-thread-count=1
httpclient.max-connections=1
With settings like these you should be able to interact with your cluster's storage using the JetS3t API library or with the Cockpit graphical tool.
To log in to your storage you will need to look up your Query interface ID and secret key on the Credentials page of the Eucalyptus web administration tool -- just hit the "Show keys" button. You will use these credentials wherever you would normally use your Amazon AWS credentials.
More Information
Here is a guide to installing and configuring Eucalyptus on Ubuntu 9.04. It's not necessarily a straight-forward process and I got stuck on a couple of steps myself, so I'm probably not the best person to ask if you run into installation difficulties.
Comments