Mashed Articles

[After a few not so technological posts, getting back to tech…]
When planning a (web) application, everyone fantasize on the tons of users that would use this ‘amazing app’. It’s great to dream but what happens if the dream comes true Assuming the architecture is scalable and was designed for large amount of users, you would still need the computing power for that (CPU, memory, DB, network…). And what happens until you reach millions of users and ‘just’ have dozens of thousands of users – how can you stretch your infrastructure to support it?
For that you need flexible scalability… I gave a try to two leading solutions:
Google App Engine – “Enables you to build web applications on the same scalable systems that power Google applications”
Amazon Elastic Compute Cloud (EC2) “A web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.”

My intermediate impression after working with both solutions for a short time is:

Google App Engine is relatively simple to use. You just need to have Google account and after a short registration you have your engine setup and free for usage until you reach ~5 million monthly page views (did not manage to test that…). It contains an ‘offline’ SDK that allows you to develop and test the application locally and then to upload it to their environment which makes it easy. The main drawback (for me) is that it is designed for running web applications using a specific engine based on the Python programming language which requires learning a bit but by using the tutorial it is still easy to use.

Amazon EC2 is actually part of a bunch of web services that allow using Amazon scalable platform for building apps by consuming processing power, storage, etc’. It’s not free and you pay for the usage (CPU time, storage, network traffic and some other parameters).Basically it allows to choose a machine image with relevant applications and to run number of instances based on the need. However although it allows using variety of machine images, containing different types of applications, it is not completely straightforward (flexibility has its cost). As it’s not free – I had to pay for my tests – up till now only 50 cents, but on the other hand, I managed only to run an existing image and even did not manage to get a proper web page shown.

Hope next time I will have more impressive results…

While enjoying holidays, I read the book “Programming Amazon Web Services” by James Murty. As explained in my earlier post, I was most interested to learn how cloud computing could be leveraged for developing integration solutions.

The book discusses 5 Amazon Web Services (AWS):

• Simple Storage Service (S3)
• Elastic Cloud Computing (EC2), virtual Linux servers on demand
• Simple Queue Service (SQS), to deliver short messages
• Flexible Payment Service
• SimpleDB – simple database with no SQL support

The book goes into quite some technical detail and has code snippets showing in detail how to interact with the Amazon services. All the samples are written in Ruby. I don’t know Ruby, but the code is quite readable (should read Enterprise Integration with Ruby some day). The author prefers the REST and the Query API. Unfortunately, he does not show anywhere the use of the SOAP API to access Amazon WS.

The 1st chapter is introductory and e.g. explains how to use self-signed certificates to connect with AWS, explains how AWS were developed for internal use by Amazon and later turned into a products, come without an SLA (except for S3) and without real support.

In the 2nd chapter, the author builds up a library of Ruby code to access the Amazon Web Services. This is very well written and gives an immediate feeling for some aspects to take into account, e.g. clock differences.

S3 is covered in chapters 3 and 4. No standard file access but the use of buckets and objects through a non-standard API (REST or SOAP); no FTP, WebDAV or SFTP. And objects cannot be modified: only deleted and re-created (after the deletion has propagated). Ruby code is shown for all the options the API offers: bucket creation/lookup/deletion, object creation/listing/deletion, ACL update/retrieval and access logging file retrieval. Tricks with HTTP header fields (object metadata), posting data through forms, alternative hostnames and BitTorrent are discussed. The last part discusses signed URI’s: this is a neat trick to make S3 resources temporarily accessible to users without Amazon account.

Chapter 4 shows some applications of the S3 service: large file transfer, backup, turning S3 into a file system (with FTP or WebDAV). Interesting to note that the author has his doubts wrt. exposing S3 as a file system. The author also discusses his own Java open source application: JetS3t. This application is a “gatekeeper” for S3 resources and authorizes local agent applications after acquiring signed URL to upload files to S3 and download files from S3.

Chapter 5, 6 and 7 dive into EC2 and how virtual Linux systems (based on Xen) can be configured using Amazon Machine Images. Ruby code is shown for every available API: keypairs (for SSH access), network security (dynamically configure the firewall), images and instances. Chapter 6 explains instances in more detail and discusses how to create new images. This involves quite some commands and scripts at the Linux command prompt. Chapter 7 discusses some sample applications: VPN server, web photo album thereby backing up data on S3. Chapter 7 also discusses issues around dynamically assigned IP addresses and the use of dynamic DNS.

The Simple Queue Service (SQS) is discussed in chapters 8 and 9. Because of the small message size, SQS is clearly meant for events with actual data stored on S3 (or elsewhere). Again Ruby code to manipulate queues and messages. Chapter 9 describes a Messaging Simulator application, not that relevant in my opinion. The 2nd application – leveraging a video conversion tool – shows how to build generic service for implementing “batch” services (Command Message pattern). The 3rd application – LifeGuard – leverages SQS to manage EC2 instance pools and dynamically scale the number of EC2 instances.

The chapter on payment service I skipped and I only skimmed through the SimpleDB chapter. Enough to learn that SimpleDB is not an RDBMS but a basic storage mechanism (no data types) with proprietary query facilities (no SQL).

The author writes fluently and gives a non-biased view on the Amazon Web Services. Sometimes the code goes into too much detail, showing how to invoke every available method of the API. Although the book is very recent (March 2008), important new features such as elastic IP addresses, persistent storage for EC2 and availability zones weren’t yet available at the time of writing. The book definitely taught me that AWS is quite proprietary and not that trivial. And to use Amazon’s cloud computing and AWS, you’d better “think like Amazon”.