« Page 11 of 18 »

  1. Amazon has announced a major update to EC2, its service that provides on-demand computing power.

    The service has graduated from "beta" to production level, which means that a Service Level Agreement is now available. If you experience uptime below 99.95% in a given month and you can provide supporting evidence, Amazon will provide a 10% service credit for that month's bill.

    Even more interesting is the news that you can now run Windows Server 2003 and SQL Server on EC2 instances. Amazon's Windows on EC2 documentation lists the prices for running these instances: from 12.5¢ per hour for a small instance with Windows only, and from $1.35 per hour for an instance with Windows, SQL Server, and Authentication Services. See the Windows AMIs listing for more details.

    Understandably, Windows instances on EC2 work very differently than the Linux ones. The folks at RightScale have published a blog post describing some of the differences and quirks of the Windows instances.

    Although I much prefer "Unixy" platforms for my own development, I can imagine situations where it would be very handy to have a Windows machine easily available -- such as for running those vital but irritating programs that are only made available for Windows. Australian Tax Office, I'm looking at you...

    To take advantage of the new Windows instances, download the latest version of the ElasticFox Firefox Extension and be sure to check out the Elasticfox Manual (PDF).

    There are comments.

  2. After building a database using an ill-judged algorithm, I ended up with junk in the database that consumed 600MB of space in the Postgres data directory. Deleting the bad data did not free this space immediately, and I was too impatient to wait for the auto vacuuming to kick in.

    Here are the steps I took to clean up the database and data directory -- with extreme prejudice.

    In the psql console:

    VACUUM FULL VERBOSE;
    REINDEX DATABASE mydatabase;
    

    On the command line:

    pg_resetxlog /path/to/my/datadir
    

    Running all of these steps is a brutal process. Do not run the pg_resetxlog command if you have important data, because you will probably want to keep your write ahead log.

    There are comments.

  3. A couple of weeks ago, a new API operation was added to Amazon's SimpleDB service: QueryWithAttributes. This operation makes it possible to retrieve all the attribute data for SimpleDB items that match a query. Previously, to retrieve item attributes it was necessary to perform a Query request, then perform follow-up GetAttributes requests for each result item returned by the service.

    In this post, I will show how to extend the SimpleDB ruby client implementation from my book Programming Amazon Web Services (O'Reilly 2008) to support this new operation, and give some usage examples.

    Let's start by adding a new method query_with_attributes to the SimpleDB Ruby sample code (available from http://examples.oreilly.com/9780596515812/). Edit the file ruby/SimpleDB.rb and add the following method definition after the existing query method. If the code below looks wonky in your browser, you can download a file instead

    def query_with_attributes(domain_name, query_expression=nil,
                              attribute_names=[], options={:fetch_all=>true})
      more_items = true
      next_token = nil
      items = []
    
      while more_items
        parameters = build_query_params(API_VERSION, SIGNATURE_VERSION,
          {
          'Action' => 'QueryWithAttributes',
          'DomainName' => domain_name,
          'QueryExpression' => query_expression,
          'MaxNumberOfItems' => options[:max_items],
          'NextToken' => next_token
          },{
          'AttributeName' => attribute_names,
          })
    
        xml_doc = do_sdb_query(parameters)
    
        xml_doc.elements.each('//Item') do |item_node|
          item = {'name' => item_node.elements['Name'].text}
    
          attributes = {}
          item_node.elements.each('Attribute') do |attribute_node|
            attr_name = attribute_node.elements['Name'].text
            value = attribute_node.elements['Value'].text
    
            if respond_to? :decode_attribute_value
              # Automatically decode attribute values if the method
              # decode_attribute_value is available in this class
              value = decode_attribute_value(value)
            end
    
            # An empty attribute value is an empty string, not nil.
            value = '' if value.nil?
    
            if attributes.has_key?(attr_name)
              attributes[attr_name] << value
            else
              attributes[attr_name] = [value]
            end
          end
    
          item['attributes'] = attributes
          items << item
        end
    
        if xml_doc.elements['//NextToken']
          next_token = xml_doc.elements['//NextToken'].text.gsub("\n","")
          more_items = options[:fetch_all]
        else
          more_items = false
        end
      end
    
      return items
    end
    

    The QueryWithAttributes operation works almost exactly like the SimpleDB service's Query operation, except it includes item attributes in the result. You can optionally specify exactly which attributes to retrieve using the attributes parameter, or leave this empty to retrieve all item attributes.

    Continuing on from the stock quote database example in my book, here is the command you would issue to retrieve the records for days on which more than 70 million Apple stocks were traded. Notice that I am taking advantage of the new sorting capabilities of SimpleDB to retrieve ordered results:

    irb> query = "['Volume' > '#{sdb.encode_integer(70000000)}'] sort 'Volume'"
    irb> sdb = SimpleDB.new
    irb> sdb.query_with_attributes('stocks', query)
    => [{"name"=>"AAPL-2007-07-26T00:00:00Z",
      "attributes"=>
    
       {"Code"=>["AAPL"],
        "High"=>[148.5],
        "Open"=>[145.91],
        "Close"=>[146.0],
        "Date"=>[Thu Jul 26 00:00:00 UTC 2007],
        "Volume"=>[78093900],
        "Low"=>[136.96],
        "Adjusted Close"=>[146.0]}},
        . . .
    

    If you were only interested in the Open and Close attribute values, you could tell the service to include only these attributes in the results:

    irb> sdb.query_with_attributes('stocks', query, attributes=['Open','Close'])
    

    It is a good idea to retrieve only the attributes you really need, because the SimpleDB service limits each response message to 1 MB and you will get more results in fewer requests if you reduce the amount of data you are asking for.

    The QueryWithAttributes operation makes it much easier to use SimpleDB because you no longer need to perform follow-up queries to retrieve attribute data. However, there may be a price to pay for this convenience so be sure to run some tests before converting your applications. It may turn out that, in some circumstances, you can retrieve results more quickly using the Query/GetAttributes approach with multiple request threads on your client, than you can using the QueryWithAttributes operation.

    In early forum discussion it looks like the new operation is both the easiest and fastest way to perform queries, but results may vary depending on how you use the service, your dataset, and your network performance.

    If you are using SimpleDB with a large data set or complex queries, try the new operation and discuss your experiences on the forum so we can gain a better idea of the strengths and weaknesses (if any) of QueryWithAttributes.

    There are comments.

  4. Amazon has announced a major new feature for their Elastic Compute Cloud (EC2) service: Elastic Block Store (EBS) persistent storage volumes.

    EBS provides for block level storage space that can be attached to your EC2 instances, but which exists separately from these instances and persists even if an instance fails or is shut down. The storage volumes can be up to 1 TB in size, can be assigned to any one of your instances on-demand, and can be easily backed up as a snapshot to S3.

    The EBS mechanism will make it significantly easier to store data reliably on EC2 instances, removing one of the main pain points of using EC2 instead of a standard data centre.

    I don't yet have much experience with EBS myself but there is a wealth of information available for EC2 users in the posts listed below, and third-party tools and libraries like ElasticFox, Typica and boto have been updated to support the storage volumes.

    There are comments.

  5. Earlier this month, Amazon changed the pricing structure for micropayment (aggregate) transactions performed with the Flexible Payments Service (FPS). FPS no longer charges a per-transaction fee for payments made from a prepaid or postpaid instrument. Fees are only changed when money is transferred into an instrument: to fund a prepaid instrument, or to settle a postpaid instrument.

    The new fee structure makes the FPS micropayment instruments much more attractive. Using the prepaid and postpaid instruments, you can aggregate many small payments into a few large transactions to minimise the cost of transaction fees over these payments. Previously, Amazon took a cut of each individual payment as well as imposing transaction fees to fund and settle the instruments, but that extra fee burden has been removed.

    There are comments.

« Page 11 of 18 »