Contemplating the brave new present
An interesting post with some drawbacks of cloud computing and EC2, from those in the trenches: Real-world cloud computing.
There are some real gems here, such as:
- [They all] used Amazon services, and most if not all of them seemed to use RightScale to manage them.
- Cost: cloud is more expensive than real machines. Cloud is good for elastic computing, not for high constant demand.
- You need monitoring services external to your cloud!
I recently installed the excellent IPython program, a beefed-up Python console that provides a raft of extra features over the default interpreter and makes it even more of a pleasure to work with this language.
When you install IPython on Mac OS X Leopard using the standard method, it only installs against the system's default version of Python: 2.5.1. However, since I had previously installed Python version 2.6.1 on my system I wanted IPython to work with this newer release.
It was surprisingly difficult to find out how to achieve this, so in
case anyone else wishes to do the same here's the process that worked
for me. Download the IPython tarball from the distributions
directory (e.g. ipython-0.9.1.tar.gz), extract the archive, change into
the extracted directory and run:
sudo python2.6 setup.py install
Notice that the command explicitly invokes the 2.6 version of python
with the python2.6 alias: this simple step is enough to
properly link your IPython installation with the newer Python. It is
obvious in hindsight that this would work, but I wasted enough time
pointlessly messing with environment variables and paths that I thought
it was worth a blog post.
Don't try this with the bleeding-edge Python 3K because IPython is not yet compatible with this version, but it seems to work fine with 2.6.1.
There are comments.
A major barrier to moving your data to an online storage location like Amazon's S3 can be the time it takes to push large numbers of bytes through your upstream Internet connection. While your connection may be fast enough to keep your data fresh and in-sync from day to day, it can be painful to do the initial data load if you have huge files, very many smaller files, a slow connection, or some combination of these factors.
I feel your pain. For the longest time I risked losing all my precious music files because I didn't want to flood my home Internet bandwidth for the four whole days it would take to upload them all.
Amazon is aiming to address this issue with the new AWS Import/Export
service, currently in limited beta in the United States.
AWS Import/Export is a pragmatic low-tech solution to the problems
of uploading huge amounts of data. You save your data to one or more
physical hard drives, then ship these drives directly to Amazon where
their contents will be read and copied into your own S3 account. When
the data is loaded, Amazon will ship the drive back to you.
You provide data loading instructions and authenticate your hard drive by sending a manifest file to Amazon in advance, then include a digitally signature of this manifest on the drive itself. Not all drive types are supported, so be sure to check the list of supported devices before you ship your drive.
The time to load your data into S3 will vary greatly based on the speed of your drive and the amount of data, but it is worth noting that this service is not yet a solution for urgently loading data. Amazon will generally only start reading your drive the next business day after it arrives, so you should expect to wait at least a couple of business days from sending your drive until your data is available in S3. The service's home page contains a handy table that gives guidance about when AWS Import/Export is likely to be faster than using the series of tubes.
The service costs $80 US per drive processed, with additional fees of $2.49 per hour the service spends reading the drive plus the standard S3 request and storage fees. Because the data is loaded over Amazon's internal network instead of the Internet, there are no data transfer fees.
Check out the service announcement blog post for pointers to tools that work with the new service, and to a calculator for comparing costs for Internet vs. physical data loads.
Although this new service is named Amazon Import/Export, only the first part of that duo is currently available. There is not yet any way to export data from S3 on physical drives. This feature is obviously in the works, but there is no timetable for its release.
There are comments.
Amazon recently released three major new features for their Elastic Compute Cloud (EC2) service -- New Features for Amazon EC2: Elastic Load Balancing, Auto Scaling, and Amazon CloudWatch. These beta services are immediately available to anyone with an EC2 account and server instances located in the US (sorry EU folks, they are US-only for now).
Amazon CloudWatch is a monitoring service that records resource and performance metrics for any EC2 instances you associate with the service, at a cost of 1.5¢ per monitored instance per hour. In addition to providing up to 2 weeks of monitoring data to EC2 users, this service also underpins the other two new EC2 services.
Auto Scaling is a service that will automatically start or stop EC2 instances on your behalf based on conditions you specify. In other words, this service allows you to automatically scale the computing power available to your application in response to changing demand.
You control your instance pool by defining triggers that react to defined conditions such as CPU load, response latency, and the number of healthy/unhealthy instances. Auto Scaling relies on CloudWatch to supply the metrics it needs to make scaling decisions, so every instance managed or started by the scaling service must be registered with CloudWatch. Happily, there is no additional cost for using Auto Scaling beyond the CloudWatch fees.
Elastic Load Balancing (ELB) rounds out the new services by providing the ability to distribute network traffic between multiple EC2 instances. ELB routes traffic at the HTTP or TCP level to instances within or across Availability Zones, and avoids routing traffic to instances that have become unresponsive. The fee for ELB is 2.5¢ per hour for each load balancer, plus 0.8¢ per Gigabyte of data transferred through the service. You will also need to pay the CloudWatch fee for each load-balanced instance.
These features constitute a major step forward in EC2 functionality that will make it easier for many users to run applications reliably in the cloud without the need to implement their own management services. However, it is important to recognize that the services are only a first step and there are many situations where they will not provide the control, precision or cost-effectiveness you will need.
Some gotchas for the services in their current incarnation include:
www.acme.com, not top-level ones like
acme.com. It also seems to limit the range of sub-1024
ports that can be balanced to 80 and 443, and does not perform some
advanced load balancing functions like HTTP session affinity
management or HTTPS termination (HTTPS connections are supported,
but only at the TCP level).As RightScale's Thorsten von Eicken points out in his discussion of the new services, there is still room in Amazon's ecosystem for third-party companies to offer value-adding services that improve on the underlying provider's offering in terms of functionality, flexibility, price and ease of use. As Amazon extends the capabilities of EC2 these companies will need to work harder to add value. This situation may be tough for them, but the fierce competition will ultimately benefit customers and accelerate the adoption of cloud services in general.
To help you get started with the new services there is a post in the EC2 forums that succinctly lists the documentation and resources you will need.
There are comments.
JetS3t (jet-set) version 0.7.1 is now available. This is the latest version of my open source Java S3 library and application suite.
Visit the JetS3t web site to download the new version, run the updated online applications, or read the latest documentation: http://jets3t.s3.amazonaws.com/index.html
This version includes bug fixes and support for the CloudFront service's
new Access Logs feature. Access logs allow you to record the
activity within your CloudFront distributions and save the information
to log files in one of your S3 buckets. See the Amazon CloudFront
Request Logging blog post for more details.
Bug fixes:
New toolkit functionality:
Cockpit application updates:
There are comments.
There are comments.