Contemplating the brave new present
Do you use Amazon Web Services for research purposes at a university, or as part of a university course?
If so, you should look into Amazon's new AWS in Education initiative which provides AWS service credits to members of the academic community. You can apply for credit programs designed for researchers, educators, student groups, or university IT departments. Check out the AWS in Education home page, and the announcement on the AWS blog.
Updated 2009-06-22: Added settings that limit JetS3t to a single HTTP connection at a time, to work around apparent thread-safety issues in Walrus.
Eucalyptus is a relatively new but rapidly developing open-source system for running your own cloud computing clusters. There have been some exciting announcements recently around this project: it has been added to Ubuntu 9.04 as the Ubuntu Enterprise Cloud technology preview, integrated into RightScale's cloud management service, and has received some venture funding.
What does all of this mean? It is becoming easier to create your own cloud computing platform using open-source tools, and then to combine your own cloud with public services like Amazon's AWS in interesting ways.
As well as helping you manage your own cloud computing clusters,
Eucalyptus includes a storage service called Walrus that is (almost)
interface compatible with Amazon's own S3 storage service. Great! So now
you can have your own personal S3-style storage service, but surely
there should be an easier way to interact with this storage than using
the s3curl.pl tool as described in the Walrus documentation?
Happily, the upcoming version of JetS3t will be compatible with the Walrus. Goo goo ga joob.
To use JetS3t with the Eucalyptus storage service you will need version 0.7.1 -- this probably won't be available on the download page until May 8th or so, but you can grab the pre-release code from the CVS repository and build it yourself if you are keen (a java.net account is required).
Once you have the latest version, you will need to configure JetS3t to
talk to your own Walrus service by editing the jets3t.properties file
in the configs directory. In summary, you must tell JetS3t where your
Eucalyptus cluster service is located and turn off the HTTPS and
DNS-naming features that are not yet supported by Walrus.
Here are the non-default settings I used to talk to a Eucalyptus front-end running on my own server with the IP address 192.168.1.128:
s3service.s3-endpoint=192.168.1.128
s3service.s3-endpoint-http-port=8773
s3service.s3-endpoint-virtual-path=/services/Walrus
s3service.https-only=false
s3service.disable-dns-buckets=true
Depending on the version of Walrus you are using, you may need to apply some additional settings. As of June 2009, there is some evidence that Walrus is not entirely thread-safe and will behave badly if you perform multiple simultaneous requests. You can work around this issue while still taking advantage of JetS3t's multi-threaded tools by configuring JetS3t to use only one HTTP connection at a time:
s3service.max-thread-count=1
s3service.admin-max-thread-count=1
httpclient.max-connections=1
With settings like these you should be able to interact with your cluster's storage using the JetS3t API library or with the Cockpit graphical tool.
To log in to your storage you will need to look up your Query interface ID and secret key on the Credentials page of the Eucalyptus web administration tool -- just hit the "Show keys" button. You will use these credentials wherever you would normally use your Amazon AWS credentials.
Here is a guide to installing and configuring Eucalyptus on Ubuntu 9.04. It's not necessarily a straight-forward process and I got stuck on a couple of steps myself, so I'm probably not the best person to ask if you run into installation difficulties.
There are comments.
Other AWS news that I haven't yet mentioned...
Amazon has announced a white paper: Creating HIPAA-Compliant Medical Data Applications with Amazon Web Services.
My JetS3t Java library is mentioned in the blog post among the tools that allow you to easily encrypt data that you store in S3. Check out the JetS3t website or discussion forum to find out more about transparently encrypting the data you store in S3.
The SimpleDB service has been updated with a new BatchPutAttributes operation that allows you to store up to 25 items with a single HTTP request.
Reducing the number of requests used to upload data can significantly speed up write operations, up to two times according to a benchmark mentioned in the announcement. You can read about the new API operation here: BatchPutAttributes.
There are comments.
Amazon's SQS messaging service has been updated with some interesting new features, as announced in the AWS blog post Powerful New Amazon SQS Features.
The original release of SQS allowed users to set access permissions for their message queues so that third-parties could participate in their messaging system. This feature was deprecated by Amazon in the 2008-01-01 version of the service's API due to unspecified shortcomings in the way the access was handled. Since then, users who relied on queue sharing capabilities for their applications needed to continue using the older API which was due to expire on May 6 this year.
In the new SQS API version 2009-02-01, the service's ability to share queues has been reinstated along with a greatly improved access control model. The Shared Queues feature allows you to grant third-parties access to your message queues using either simple API calls to control principal-based permissions (for other AWS account-holders) or by writing more sophisticated rules using a new Access Policy Language. The policy language provides many more features, including the ability to grant public access or to limit access based on IP address ranges.
Another feature that was lost in the 2008-01-01 API update was the ChangeMessageVisibility operation, which allowed users to change the visibility timeout for a specific message on-demand as well as when the message was delivered. This feature was very useful for delaying the redelivery of a message when it turned out that your application would not be able to process the original message as quickly as expected, a common occurrence in applications that process work units with unpredictable processing times.
This feature too has been reinstated in the new API version under the same name: ChangeMessageVisibility. With this operation you can change/extend a message's visibility timeout up to 12 hours.
SQS has now joined the list of services with endpoints located in Europe, which will make the service perform better for users in that part of the world. This is particularly good news for anyone running EC2 instances in the EU region, because data traffic between these instances and EU-based SQS queues is free.
There is a lot to like in this service update. See the full list of changes, and detailed documentation, in the Amazon Simple Queue Service Developer Guide.
There are comments.
Grig Gheorghiu of the Agile Testing blog has a very good post about his experiences deploying large-scale applications on Amazon's EC2 service: Experiences deploying a large-scale infrastructure in Amazon EC2. Lots of great tips here.
There are comments.
There are comments.