« Page 12 of 18 »

  1. I am pleased to announce the release of JetS3t version 0.6.1, the newest version of my open source Java S3 library and application suite.

    This release includes bug fixes, application improvements, and support for some of the newest features of Amazon's Simple Storage Service (S3). Highlights include:

    • Support for copying, moving, and renaming S3 objects with the programming toolkit and Cockpit application
    • The metadata associated with objects can now be updated without the need to upload an object again
    • Improved data verification options for ensuring that information is not corrupted in transit
    • The Synchronize application supports new options to improve performance and decrease memory requirements when synchronizing large numbers of files.

    You can download version 0.6.1 from the JetS3t Downloads page. The web site has also been updated with the latest application manuals, code samples, and API documentation.

    Enjoy!

    There are comments.

  2. The latest version of Amazon's SimpleDB service is available, and happily it now allows for sorting of query results. This highly-anticipated feature will make the service much more useful.

    Sorting still suffers from some limitations: you can only sort by a single attribute, you cannot perform sorting in queries that contain union or not predicates, and the attribute against which you are sorting must be present in a query predicate. However, it is still a massive improvement on sorting query results in your own client code.

    For example, given the following SimpleDB query that finds historic stock records for a given time period:

    ['Code' = 'AAPL'] intersection ['Date' > '2007-12-01']
    

    With the new sort operator, you can now instruct the service to return the results in reverse (descending) date order like so:

    ['Code' = 'AAPL'] intersection ['Date' > '2007-12-01'] sort 'Date' desc
    

    Although you can only sort by an attribute that is also mentioned in a predicate, you can easily work around this restriction by including a starts-with '' predicate for that criteria, which will always evaluate to true. Here is a query to return stock records sorted according to the closing price, from lowest to highest:

    ['Code' = 'AAPL'] intersection ['Close' starts-with ''] sort 'Close' asc
    

    The SimpleDB update also includes some other new features, such as a new does-not-start-with operator and support for 10 predicates per query instead of 5, but I think it is the sorting that will have the most impact.

    There are comments.

  3. As I post this, Amazon's S3 and SQS services have both been down for hours. The latest updates to the service health dashboard are promising so things will hopefully be resolved soon, but this definitely isn't a good sight:

    The AWS Service Health Dashboard showing that the S3 and SQS services are down

    Update

    Amazon has posted a full explanation of the recent S3 outage here: Amazon S3 Availability Event: July 20, 2008.

    This makes for interesting reading, but ultimately the lesson from this most recent outage is that service failures can and will happen. Even to Amazon.

    If you are building a cloud-based application you should try to include as many contingency options as much as possible, but few contingencies could have survived that outage. Let's hope it isn't repeated.

    There are comments.

  4. When developing a web site or application, you may sometimes need to expose the development version running inside your network to the public Internet. This can be necessary to test the communication between your application and an external service, such as PayPal.

    If you have secure shell access to a publicly accessible server, you can achieve this using the port forwarding feature of ssh to redirect traffic from the Internet to your own machine. Ideally, you would use the secure shell's -R option to forward a remote port. For example, to forward traffic arriving on port 8888 of your public server to port 8000 on your local machine you would do the following:

    ssh -R *:8888:127.0.0.1:8000
    

    However, for remote forwarding to work the GatewayPorts option must be enabled on your public server. This option is rarely enabled by default, and if you lack the admin privileges or desire to change this setting you will be unable to use remote port forwarding directly.

    In this situation there is a work-around that achieves the same result without relying on the GatewayPorts setting being enabled: local port forwarding from the public server back to your own machine. Because your machine probably isn't accessible from the public server (at least, it shouldn't be...) you will need to pre-prepare an ssh tunnel that will allow you to connect back to your computer.

    All this talk of tunnels upon tunnels is confusing, so let's cut straight to the commands. First, log in to the public server with ssh while opening a port to allow a reverse ssh connection back to your machine. Here, I will forward the server's port 2222 to port 22 on my local development machine:

    ssh -R2222:127.0.0.1:22 james@publicserver.com
    

    Now, connect back to your own computer using the tunnelled port 2222, while at the same time opening a public port to direct Internet traffic from the server to your machine. The following command will forward traffic sent to the server's 8888 port back to my computer's 8000 port.

    ssh -g -L8888:127.0.0.1:8000 -p 2222 jmurty@127.0.0.1
    

    In this command, the -g option allows Internet traffic to be forwarded from the server rather than just local traffic, and the name jmurty is the user name credential for my development machine. Although I am seemingly connecting to the server at the loopback address 127.0.0.1, because I am using port 2222 I will actually be tunnelled back to the machine I started from.

    In my experience, this work-around will give you the desired results even when the GatewayPorts setting is disabled on the server. It is not an ideal solution because the tunnelled traffic is encrypted twice so it is quite slow, but this trick may be useful as a stop-gap measure until your server administrator enables this setting on the server.

    There are comments.

  5. Amazon staff member Jinesh Varia has written a white paper in two parts that describes a "cloud architecture" application built on the SQS, EC2, S3 and SimpleDB services, and discusses best practices for using these services.

    This paper is an excellent resource for anyone designing a scalable application based on AWS, especially Part 2 and the section "Tips for Designing a Cloud Architecture Application" in Part 1.

    The paper is available as web pages here:

    There is also a PDF document version that includes a bonus extra: an appendix that summarizes the differences between the three data storage services -- S3, SQS, and SimpleDB -- and explains which service is most appropriate for particular tasks.

    There are comments.

« Page 12 of 18 »