Amazon for Infrastructure-on-Demand

The buzz is building around Amazon Web Services as an application platform. Don MacAskill has been using AWS’ S3 storage service for SmugMug, and according to Jeremy Zawodny, will be talking about it at this year’s ETech conference. Jeremy and others have been experimenting with S3 as storage backup for desktop and laptop systems. Even Dave Winer is experimenting with S3.

When designing GigaVox Audio Lite (currently in by-invitation-only alpha test) I decided to use not just S3 (storage) but essentially all of AWS’ services for our infrastructure. This graphic should give you some idea of what we’re doing:

gigavox-amazon-003-old.jpg

In addition to S3 (storage) I’m using the Elastic Compute Cloud (EC2) which allows us to instantiate servers on-demand. If we need a dozen extra servers for two hours of transcoding, we just fire them up for as long as we need them (perhaps a few hours) then shut them down. And that can be automated based on the load on the current servers. Dynamic computing power on demand. The dream has been realized.

To communicate among our servers and with the external systems we also make heavy use of the Simple Queue Service (SQS). There are actually many more queues than shown in the diagram which is somewhat out of date.

In asking around, it appears we have one of the most complex and sophisticated systems built to date using all three AWS services. It is infinitely scalable, extremely reliable and costs very little until we need it. And our system administrator doesn’t have to carry a pager. All those servers are someone else’s problem. No hardware or software to buy, maintain, upgrade, etc. It has been a lot of fun to actually make use of some of the asynchronous message-based architecture ideas in my book, Loosely Coupled. Hey…this stuff actually works! Sweet. Congratulations to Amazon for brilliant vision and execution.

If you want to see it in action for the production and publishing of your podcast, send me an email, tell me about your podcast, and I’ll send you back an invitation to the alpha test.

24 thoughts on “Amazon for Infrastructure-on-Demand

  1. “all three AWS services…”

    Amazon offers seven services, not three. You left out Amazon E-commerce Service, Amazon Mechanical Turk, Alexa, and Amazon Historical Pricing service..

    Like

  2. I’m myself using S3 for quite some time — other than a stray slow response time once or twice, the service has been amazing. And even if the resp time is slow, Amazon folks are very quick to rectify it.

    the amazon WS is just brilliant and a life-saver when it comes to startups who want to focus on execution and business rather than the headache of dealing with storage, backups and stuff like that.

    slideshare.net also uses amazon s3 to host the flash based ppts.

    Like

  3. Pingback: Geek News Central
  4. Awesome setup, Doug. You’re one of the few out there really taking full advantage of Amazon’s offerings. For the curious, Basejumpr.com is another startup using S3, SQS, and EC2 in conjuction. I can’t wait to see what else gets built on top of Amazon’s infrastructure.

    Like

  5. Pingback: berbs.us
  6. I use S3 too for streaming videos over HTTP. you can check out the servcie at KONKAN TV. Apart from the ocassional failed SOAP / REST file push requests and ocassional CPAINT errors and/or streaming speeds, it’s been an awesome ride so far…

    Like

  7. Pingback: Cantoni.org
  8. Hi Doug, i was listening to your video on Amazon Web Services. Thanks for sharing the info.

    I wanted to follow up on your decision to use the external Database server. Can you provide some detail on why you did not choose to use one of the images as a database server.

    regards

    Like

  9. Hi, I have a question about the real world usabitity of the Amazon Web Services. Could I run windows servers in windows vm clusters on the Amazon Elastic Compute Cloud (Amazon EC2) Windows Server running SQL, Office Middle Ware, VOIP… like a real hosted utility grid?
    Can’t find any data on people using windows server os’s with it?
    Thanks

    Like

  10. Hi there, the graphic doesn’t show up any more😦
    Any way I can have a glance at it? I’m really interested in how it works with Amazon s3!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s