"Monitoring cloud assets, scaling capacity automatically, and balancing traffic efficiently have been among the most requested Amazon EC2 features from our customers," said Peter DeSantis, General Manager of Amazon EC2. "Together, these capabilities provide customers more control of their AWS resources and enable them to architect for even better performance, resilience and cost savings."

CloudWatch gives Amazon customers visibility into resource use, operational performance, and overall demand patterns, including metrics such as CPU use, disk reads and writes, and network traffic. Auto Scaling ensures EC2 usage increases during traffic spikes to maintain performance and decreases during lulls to reduce costs, making it particularly appropriate for apps with frequent use fluctuations. Elastic Load Balancing allows for fault tolerance in applications, detects unhealthy instances within a pool, and reroutes traffic to healthy instances until the unhealthy instances have been restored.

Amazon hopes these features will improve application performance, lower costs, and make life easier for developers and entrepreneurs. Amazon CloudWatch and Elastic Load Balancing are available on a pay-as-you-go basis, and Auto Scaling is enabled by Amazon CloudWatch and carries no additional fees. Features are currently available in the U.S. and should be available in the EU shortly.

To us, this looks like a great way to introduce students to cloud computing, and, as far as we are concerned, the more students get access to this technology, the better. We also like Amazon's idea to grant researchers access to its services, as this can mean significant cost reductions for many academic projects, which, given the current economic climate, can only be a good thing as well - and, of course, it will also introduce these researchers to the potential of cloud computing.

The way MapReduce works is a fairly straightforward concept: You take a problem that requires working with a giant (and we're talking massive - sometimes petabytes) dataset, distribute working with the dataset over thousands of separate processes (called mapping) and then taking the thousands of results you get back and reducing those results into a single master result. For certain tasks, MapReduce can vastly improve the efficiency of these types of tasks, and adding more computing power gives you a linear improvement in speed.

Yahoo! has been using its own version of Hadoop for a while now. And even before this offering, larger Amazon Cloud Computing customers have already begun to use Hadoop in EC2. This is from Wikipedia's article on Hadoop:

As an example The New York Times used 100 Amazon EC2 instances and a Hadoop application to process 4TB of raw image TIFF data (stored in S3) into 1.1 million finished PDFs in the space of 24 hours at a computation cost of about $240 (not including bandwidth).

As Amazon says on its blog, "After a while [developers] tend to report that they begin to think in terms of the new style, and then see more and more applications for it." Which we believe means that MapReduce is the new, big hammer, and as developers start looking around, every dataset starts looking like a nail. This is good news for Amazon as it only stands to profit.

Currently, you can only manage your EC2 instances through the new console, but over time, Amazon will give users the ability to mange other AWS services like its S3 storage service.

Ease of Use

The new console makes it especially easy for first-time users to set up their instances on EC2. While you can choose from a bewildering array of community-created instances (think custom Ubuntu mail servers), Amazon will present you with a sane amount of 'quick start' instances and relatively conservative security settings.

Amazon also created a great screencast that walks you through creating your first EC2 instance. While using Amazon's web services was already pretty manageable thanks to tools like Elasticfox, this new console gives users an easy way to control their instances, while also allowing new users to start using EC2 without too much hassle. If you always wanted to experiment with EC2, but were intimidated by the set-up process, now would be a good time to give it a try.

Called AWS Hosted Public Data Sets, the service will enable you to use public data within your Amazon EC2 environment. Select public data sets will be hosted on AWS for free as an Amazon EBS snapshot.

However, as there is no indication that the Gutenberg Project will be added to AWS, we've calculated how long it would take to download and upload the 80GB UGI Virtual Conformer Library, one of the listed data sets AWS plans to host.

Using a residential cable provider in California, it would take 22 hours 36 minutes to download, and 3 days 36 minutes to upload to a server in the same state. However, if the server was in New York and we accessed it from California, it would take 3 days 42 minutes to download, and 7 days 14 hours to upload. Clearly inefficient.

People have been searching for better ways to access public data sets for some time, and AWS Hosted Data Sets may just be the answer they've been looking for; allowing anyone to do the type of computing that in the past has been limited to large organizations with lots of money.

Current data sets that Amazon are working on include: annotated Human Genome data, PubChem and UGI Virtual Conformer libraries, the U.S. Census, various labor statistics, and various economic and transportation databases.

AWS will continue to add to the collection over time, and this is where you come in.

If you have a public data set and hold the rights to the distribution of it, you can submit a request on the AWS Public Hosted Data Sets site to have it included.

This is huge.

For Yieldex - a company that predicts available online advertising inventory to help optimize campaigns - that creative funding angle came in the form of participating in the Amazon Web Service Start-Up Challenge. And it paid off, twofold. Not only did they end the week with an extra $100,000 by winning the second annual Amazon AWS Start-Up Challenge, they also - as we reported in Jobwire - managed to woo a former AOL exec to join the team.

Clearly, the judges weren't the only ones who found the solution compelling. The former Senior Vice President for Network Development at AOL's Platform A, Larry Allen, has joined the Yieldex executive team as President. Allen joins a team that has its roots in Matchlogic and NetGravity, two online advertising oriented companies from the dotcom days.

Yieldex follows in the footsteps of Ooyala, the winner of the inaugural contest. We reviewed Ooyala last year about this time.

What Yieldex actually does or how it really helps advertisers optimize their inventory remains a bit cloudy, but we'll be sure to keep an eye on their progress.

In this era everything is different - the stack, the players and the potential for value creation. Let's outline the basic shape of this emerging era, in particular defining what makes up the new stack.

At the Top - SaaS: these are the end user services that we actually interact with, such as Basecamp. This is the "final mile". This is what we used to call application software, vertical systems or value added systems. Although SaaS is sometimes also used to describe the layer below, we prefer to label the top as SaaS and the middle as Cloud Computing. Typically this layer has had thousands of companies. These are our bootstrapped Gritty Entrepreneurs.

In the Middle - Cloud Computing: this is the Cloud where we witness the "sound and fury" of BigCos battling it out - Amazon, Google, Microsoft, IBM and others. This layer is the most fluid and where all the deals are. This layer can be seen as two layers, but the difference is very blurry. Some SaaS companies create some "middleware" that they position in this layer. Some start-ups create middleware as their primary focus, with an end game of getting acquired by one of the Cloud BigCos. Over time, these will tend to get rolled up into a few big platforms that compete by providing higher levels of abstraction for developers.

At the Bottom - Core Technology: this is what we might call "traditional Silicon Valley", hard core patent-protected technology sold to big companies that use it as part of their stack. Arista, the latest venture from Andreas Bechtolsheim falls into this category.

Spectators And Players

Most of us are spectators in the Cloud Computing game. It is fun to watch the big guys duke it out and ReadWriteWeb will continue to report on that. Entrepreneurs need to understand the strategies of the big players who will be their "platform partner". But we all have lots of opportunities to be players at the top of the stack, in the SaaS layer. This is where there are low barriers to entry, massively reduced R&D costs and incumbents who will be slow to embrace SaaS for fear of cannibalizing their core business.

Has The Stack Value Inverted?

Traditionally, value was at the bottom of the stack, which is why Microsoft and Intel were so dominant in the past. With a few notable exceptions like SAP, the top of the stack tended to be smaller companies.

It is possible that this has inverted, that the real value is now at the top of the stack and not at the bottom. For example, Arista will probably be very successful, but their market will be limited to the few companies who build huge data centers. Those clients will place huge orders but will also have a lot of negotiating clout.

Lock-In And Network Effects?

So maybe the value is all in the middle now? This is certainly where all the action is today. The two big questions at this layer are:

1. Lock-in? How easy will it be to move your SaaS service between Amazon AWS, Google App Engine, Microsoft Azure and other contenders? Today there is quite a lot of technical lock-in, you cannot move from one to another without some re-coding. But is that a big deal? No, because a) any Platform that jacks up prices will get hammered by their competitors and b) when you do need to move, it may require some coding changes but the move is transparent to end users. So, very little lock-in.

2. Viral Network Effects? Market leaders will get a lower cost of sale, but there is no social media viral effect at the Cloud Computing Platform layer.

Without Lock-In or Viral Network Effects, this layer will be commoditized. It will be very, very big but it will be a thin margin commodity business that is all about scale.

So The SaaS Cream Floats To The Top?

This is our theory. The value is with the small SaaS companies in the "final mile" interacting with end users. This is what we are seeing with our bootstrapped Gritty Entrepreneurs.

What About Players Across Layers?

Next week I will be at Dreamforce, the Salesforce.com annual event in San Francisco. Salesforce.com is the SaaS pioneer that defined the market. At some stage they decided that being on the top layer only was not enough and they created their Force.com "platform" on which others could create applications.

IBM also operates at many layers of the stack. But they do so with separate divisions that would be large companies in their own right.

It will be interesting to see how this stack evolves and specifically how well Salesforce.com succeeds with their mission to operate at both the top and the middle layer.

Some companies just blog about updates to their own technology and that's good for existing users to see. Others are fun to read whether you're a user or not. Here are some of the company blogs we recommend reading for a good time.

You can't talk about interesting company blogs without mentioning Signal vs. Noise, the wildly successful blog about design, usability and small business from the makers of project management service Basecamp. This blog could easily stand on its own as compelling reading even if there wasn't a company behind it selling services. Sure enough, it's even got an ad on it from the elite boutique ad network The Deck. Signal vs. Noise has 88k subscribers - making it fun and educational for those subscribers is great for the business of 37signals.

Ning

Ning powers niche social networks and the company's blog is a great place to find out about all kinds of groups that are actively using this technology. Some of them a real surprise. The International Society of Space Entreprenuers, the Eat Local Foods Coalition of Maine and the ASPCA Book Club all have social networks! Who knew? Now how about some data portability, Ning? If you want to read about the theoretical side of niche social networks, check out data portability lovin' competitor PeopleAggregator's blog, written by the company's prolific CEO Marc Canter.

Viddler

Viddler is a crazy feature rich video hosting service with a very good blog. In addition to feature announcements, the company posts a wide variety of videos that are interesting to watch. That's where I discovered this one below, for example.

Newsgator

RSS company Newsgator has a great blog about RSS use cases, data portability debates and other industry topics. Maybe I'm just an RSS head - but I really enjoy their blogging. Newsgator competitor Attensa also has a very good blog about all things RSS - but it sure could use some more updates!

A.viary

Collaborative design tool A.viary has a couple of very good blogs. Their idea blog is full of interesting content that regularly hits the front page of digg. These folks have some seriously juvenile gender issues going on, which I have given them a hard time about before, but their blog is still fun to read otherwise. Every post they put up is worth a look.

Adaptive Path

User experience and design firm Adaptive Path publishes a very thought provoking blog. I usually scan company blogs for announcements and videos - but their longer posts often convince me to stop and read. OLPC: The Beauty of Failure and Greedy Mobile Interfaces? I'll stop and read posts with titles like that.

Amazon Web Services

What could be more dreary than commoditized data processing and storage services? While that might sound boring, the AWS Blog does a great job of highlighting cool things that are done on top of Amazon Web Services. Lately they've put up posts about how AWS are being used by the New York Times to provide online access to 150 years of archives and by the little ShareThis widget that you've probably seen on hundreds of thousands of blogs around the web. Lots of charts, graphs and other fun stuff for nerds on this blog.

Strands

Here at RWW we've been following the mega-funded recommendation service Strands for several years. We find what they do fascinating. Their company blog is mostly about company announcements, but they have really interesting announcements. Last.fm on Nokia phones? Cool! The most interesting section of their blog though is the data portability category and throughout the blog you'll find some really deep thoughts on cutting edge innovation.

Articulate

Articulate is a an e-learning tools company with a very popular blog. More than 20,000 subscribers actively discuss topics like how to make a good screencast and how best to work with clip art. This was a new one to me but I've already enjoyed spending some time on this company's blog. What more could you ask from a company blog? Good content creates a community of advocates that share the blog with friends, some of whom undoubtedly will purchase the company's products.

These Are But a Few

There really are a fair number of interesting company blogs around the web. We'd love to hear about some of your favorites. Others that are worth checking out include Oracle Apps Lab for a fun discussion of web 2.0 in the enterprise, the Lijit blog on "searching the social graph" and the many fun blogs published by Adobe.

If you found this post of interest, you might enjoy reading our coverage of some of the top new social media company evangelists as well.

We hope you find some of these blogs worth subscribing to and we'd love to learn about other blogs that cover topics of general interest that even non-customers would enjoy reading.

"Increasingly, we see that organizations of all sizes are putting AWS to use in new, innovative, and mission-critical ways," wrote Amazon Web Services Evangelist Jeff Barr in a blog post. "These organizations have told us that they need a more direct and more discreet way to request assistance and to report problems."

The new services don't come particularly cheap, costing $100 per month or $0.10 per dollar of total monthly usage for silver level (whichever is greater), or the greater of $400 per month or $0.10-$0.20 per dollar of total monthly usage for gold level.

Amazon is also beefing up support options for free customers with the release of the new AWS Service Hearth Dashboard that monitors the status of all AWS services. Amazon says that during outages, users can expect to see updates from the team every 15-30 minutes until things are fixed. Status updates can be accessed via the page or by RSS.

If Amazon really wants to upgrade their free support center, we suggest they build something off the excellent Get Satisfaction service, which just released an API yesterday. Some of Amazon's web services already have a page on Get Satisfaction, but the company has yet to send over any official support reps.

Some wondered after Google's App Engine announcement Monday evening when Microsoft would offer a competing cloud computing platform. The biggest tip off from the job advertisement that Red Dog is it, is that the CIS team wants the platform to "lead the marketplace as the best platform for rapid development, deployment, and maintenance of internet services and applications." Microsoft will supposedly roll out a first version of Red Dog to "external customers" (defined later as "ISV customers who are ... early adopters") this year.

As Kniskern points out, not much is known about Red Dog at this point, but indications seem to point to some sort of platform as a service offering from Microsoft dropping within the next year.

Note: No, that's not a real Red Dog logo. It's just Clifford The Big Red Dog with a tiny Windows Live logo dangling from his collar...

Last night Amazon announced the newest addition to the Amazon Web Services suite: Amazon Fulfillment Web Service (AFWS).

Amazon has offered other businesses access to its fulfillment infrastructure for some time through the Fulfillment by Amazon service, but today's announcement means that the whole process will be automated. It's a webservices world!

As we discussed in our recent post on APIs and Developer Platforms: The Pros and Cons, an API is like an invitation to start a structured business development relationship on the fly, without lengthy technical or political negotiations. The Fulfillment API is what brings the Amazon Fulfillment Services to life.

Amazon Web Services have enabled a whole new class of web startups to offer storage and processing features far beyond what they could have in-house. At least in terms of bandwidth use, those webservices are now bigger than Amazon.com proper - itself one of the most visited sites on the internet.

We'll see if Amazon Fulfillment can do the same to small shops selling physical goods. The centralization of the infrastructure could be very interesting. Perhaps Walmart will buy Amazon someday for its webservices. I look forward to reading what Nick Carr has to say about this.

So far only Amazon of the big web companies have opened their hosting architecture up to outsiders, but a number of smaller players have tapped into the growing market for cloud based hosting solutions. Mosso's platform seeks to match the scaling power of a compute cloud with the ease and simplicity of a shared hosting environment.

Unlike competitors such as Joyent or Amazon, Mosso's system does not offer customers root level access to their servers. Instead, servers are preconfigured with a range of software options and are fully managed similar to a shared hosting environment. Keeping their hosting platform standardized is what allows them to easily monitor and scale the service as needed. For example, last week Mosso experienced a significant spike in load on its PHP cluster and added 10 servers without customers noticing, company executives told me.

Today, Mosso is announcing a new payment scheme that they feel is an industry first for a cloud computing platform. According to Mosso, scaling on Amazon or other compute cloud options means adding more instances, which means you're still paying on a server by server basis. Mosso's new pay structure is based 100% on requests, which they feel is more accurate in terms of charging you only for the resources you use.

Mosso's pay structure starts with a base fee of $100 per month, with a rate of $0.25 per gigabyte of bandwidth, and $0.50 per gigabyte of storage. They also charge $0.03 per 1000 requests with 3 million included. Being charged per gig on bandwidth and storage is a familiar pricing structure for anyone on shared hosting or a managed server.

The company is also announcing an updated control panel and will soon roll out a MySQL backup utility that will take automatic snapshots of databases down to the table level. That sort of utility wouldn't be possible on EC2, according to Mosso, because every instance is configured differently.

Note: Due to a miscommunication on our part we briefly ran this story yesterday on our web site when in fact the new Mosso pricing plan was not going live until today (February 19). We apologize for any confusion this caused.

Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com’s global websites combined.

As TechCrunch's Erick Schonfeld points out, "That means startups and other companies using Amazon’s Web-scale computing infrastructure now bigger collectively than Amazon.com, at least as measured by bandwidth usage."

Amazon also announced that there are now over 330,000 developers registered to use Amazon Web Services (AWS), an increase of 30,000 developers in the last quarter. Adoption of Amazon's platform is very likely helped by the over 200,000 developers who have signed up to use Facebook's platform. Amazon recently launched a specialized page for Facebook developers in an effort to court app developers to the AWS platform.

In November, Robert Scoble was in disbelief about the prospect of a serverless Internet company. Because of Amazon's web services, anyone can build a scalable web site without actually owning a single server -- and many companies are utilizing the service to do just that. "Let’s get this straight. Amazon used to be a book store," he wrote. "Now they are hosting virtualized servers for Internet companies."

A little over a year ago, Alex Iskold wrote about the Amazon web services stack, saying that it was "evidence of a new computing paradigm, where web services in aggregate give rise to a new web-based operating system." But Alex also conceded that small and medium sized businesses would be the first to get on board with Amazon's web scale computing platform. "Wall Street is not going to jump on this," he wrote.

Amazon's AWS success stories page still doesn't include any fellow members of the Fortune 500, but it does include a number of successful or well-funded web 2.0 start ups. The usage numbers released today indicate that even though Amazon isn't making a ton of money from their web services (TechCrunch guesses that AWS revenue was part of the $131 million "Other" category for the 4th quarter -- which includes other services and isn't much of the $5.7 billion in total Q4 revenue), they are striking a chord with developers and positioning themselves as a major provider of infrastructure services for an entire generation of web applications.

DevPay offers a lot of flexibility for developers, allowing users to create their own pricing plan "using any combination of one-time charges, recurring monthly charges, and metered Amazon Web Service usage." Beyond automatically determining usage, sending out bills, and handling payment processing, DevPay also handles the courtesies such as email notifications -- all of which are customizable via the API.

Amazon's first customer for their new service is Red Hat, who is offering Red Hat Enterprise Linux on Amazon EC2 via DevPay.

There was some initial confusion about the difference between DevPay and Amazon's Flexible Payment Service, which allows web sites to receive payments. "I'm still not clear on what the difference is from Amazon FPS. Is it specifically the ability to meter AWS usage? If so, why not simply build this into FPS?" wrote commenter A. Logan Murray on Amazon's announcement post.

The difference, according to Amazon, is that while FPS allows developers to charge for any sort of service, DevPay specifically hooks into the AWS. When you use DevPay to charge users for S3 usage, for example, the payments are made directly to Amazon and then deducted from your S3 bill (with the remainder being deposited into your Amazon account). DevPay cuts out the middleman -- your customers are paying Amazon directly for the web services they are using.

DevPay is priced at $.30 per transaction + 3% of the total amount billed. That's a bit higher than PayPal, but because it can also do the legwork of calculating how much your customers owe you based on how much they use, DevPay should be a very attractive way for developers to charge customers for AWS usage. Like other Amazon services, this one is only available in the US right now.

Yet, even from a casual glance, it is clear that Amazon's work is very important. Since early last year, the e-commerce giant has been making forays into becoming a Web OS company. Amazon has been methodically exposing pieces of its own infrastructure as commodity web services, and in the process confusing Wall Street analysts and making thousands of startups quite happy.

Dr. Vogels has been both the architect and the evangelist of this effort. In his speech at last year's ETech conference he explained that by leveraging the Amazon Web Services Stack, web developers are finally able to focus on the core business logic of their apps and services. Hiding the enormous complexity of building a scalable web business behind a simple API, Amazon is paving the way toward a whole new web ballgame.

What Vogels talked about on his blog a few weeks back is not a public web service, but a piece of internal infrastructure, which allows Amazon to service millions of customers. The paper is a unique revelation about the inner workings of one of just a handful of Internet giants. It is also a preview of the web services to come in the next decade. In this post we take a close look at Dynamo, discuss where it fits and consider its implications.

Scalability Issues With Relational Databases

Before discussing Dynamo it is worth taking look back to understand its origins. One of the most powerful and useful technologies that has been powering the web since its early days is the relational database. Particularly, relational databases have been used a lot for retail sites where visitors are able to browse and search for products. Modern relational database are able to handle millions of products and service very large sites.

However, it is difficult to create redundancy and parallelism with relational databases, so they become a single point of failure. In particular, replication is not trivial. To understand why, consider the problem of having two database servers that need to have identical data. Having both servers for reading and writing data makes it difficult to synchronize changes. Having one master server and another slave is bad too, because the master has to take all the heat when users are writing information.

So as a relational database grows, it becomes a bottle neck and the point of failure for the entire system. As mega e-commerce sites grew over the past decade they became aware of this issue - adding more web servers does not help because it is the database that ends up being a problem.

Dynamo - A Distributed Storage System

Unlike a relational database, Dynamo is a distributed storage system. Like a relational database it is stores information to be retrieved, but it does not break the data into tables. Instead all objects are stored and looked up via a key. A simple way to think about such a system is in terms of URLs. When you navigate to the page on Amazon for the last Harry Potter book, http://www.amazon.com/gp/product/0545010225 you see a page that includes a description of the book, customer reviews, related books, and so on. To create this page, Amazon's infrastructure has to perform many database lookups, the most basic of which is to grab information about the book from its URL (or, more likely, from its ASIN - a unique code for each Amazon product, 0545010225 in this case).

In the figure above we show a concept schematic for how a distributed storage system works. The information is distributed around a ring of computers, each computer is identical. To ensure fault tolerance, in case a particular node breaks down, the data is made redundant, so each object is stored in the system multiple times.

In technical terms, Dynamo is called an eventually consistent storage system. The terminology may seem a bit odd, but as it turns out creating a distributed storage solution which is both responsive and consistent is a difficult problem. As you can tell from the diagram above, if one computer updates object A, these changes need to propagate to other machines. This is done using asynchronous communication, which is why the system is called "eventually consistent."

How Dynamo Works

The technical details in Vogels' paper are quite complex, but the way in which Dynamo works can be understood more simply. First, like Amazon S3, Dynamo offers a simple put and get interface. Each put requires the key, context and the object. The context is based on the object and is used by Dynamo for validating updates. Here is the high level description of Dynamo and a put request:

• Physical nodes are thought of as identical and organized into a ring.
• Virtual nodes are created by the system and mapped onto physical nodes, so that hardware can be swapped for maintenance and failure.
• The partitioning algorithm is one of the most complicated pieces of the system, it specifies which nodes will store a given object.
• The partitioning mechanism automatically scales as nodes enter and leave the system.
• Every object is asynchronously replicated to N nodes.
• The updates to the system occur asynchronously and may result in multiple copies of the object in the system with slightly different states.
• The discrepancies in the system are reconciled after a period of time, ensuring eventual consistency.
• Any node in the system can be issued a put or get request for any key.

So Dynamo is quite complex, but is also conceptually simple. It is inspired by the way things work in nature - based on self-organization and emergence. Each node is identical to other nodes, the nodes can come in and out of existence, and the data is automatically balanced around the ring - all of this makes Dynamo similar to an ant colony or beehive.

Finally, Dynamo's internals are implemented in Java. The choice is likely because, as we've written here, Java is an elegant programming language, which allows the appropriate level of object-orineted modeling. And yes, once again, it is fast enough!

Dynamo - The SLA In A Box

Perhaps the most stunning revelation about Dynamo is that it can be tuned using just a handful parameters to achieve different, technical goals that in turn support different business requirements. Dynamo is a storage service in the box driven by an SLA. Different applications at Amazon use different configurations of Dynamo depending on their tolerence to delays or data discrepancy. The paper lists these main uses of Dynamo:

Business logic specific reconciliation: This is a popular use case for Dynamo. Each data object is replicated across multiple nodes. The shopping cart service is a prime example of this category.
Timestamp based reconciliation: This case differs from the previous one only in the reconciliation mechanism. The service that maintains customers' session information is a good example of a service that uses this mode.
High performance read engine: While Dynamo is built to be an "always writeable" data store, a few services are tuning its quorum characteristics and using it as a high performance read engine. Services that maintain a product catalog and promotional items fit in this category.

Dynamo shows once again how disciplined and rare Amazon is at using and re-using its infrastructure. The technical management must have realized early on that the very survival of the business depended on common, bullet proof, flexible, and scalable software systems. Amazon succeeded in both implementing and spreading the infrastructure through the company. It truly earned the mandate to then leverage its internal pieces and offer them as web services.

How Does Dynamo Fit With AWS?

The short answer is that it does not, because it is not a public service. The short answer is also shortsighted because there are clear implications. First, is that since Amazon is committed to building a stack of web services, a version of Dynamo is likely to be available to the public at some time in the future.

Second, Amazon is restless in its innovation; and that applies to web services as well as it applies to its retail business. S3 has already made possible a whole new generation of startups and web services and Dynamo is likely to do the same when it comes out. And we know that more is likely to come, as even with Dynamo, the stack of web services is far from complete.

Finally, Amazon is showing openness - a highly valuable characteristic. Surely, Google and Microsoft have similar systems, but Amazon is putting them out in the open and turning its infrastructure into a business faster than its competitors. It is this openness that will allow Amazon to build trust and community around their Web Services stack. It is a powerful force, which is likely to win over developers and business people as well.

The Future - Amazon Web OS

To any computer scientist or software engineer to watch what Amazon is doing is both awesome and humbling. Taking complex theoretical concepts and algorithms, adopting them to business environments (down to the SLA!), proving that they work at the world's largest retail site, and then turning around and making them available to the rest of the world is nothing short of a tour-de-force. What Emre Sokullu called HaaS (Hardware as a Service) in his post yesterday is not only an innovative business strategy, but also a triumph of software engineering.

Amazon is on track to roll out more virtual web services, which will collectively amount to a web operating system. The building blocks that are already in place and the ones to come are going to be remixed to create new web applications that were simply not possible before. As we have written here recently, it is the libraries that make it possible to create giant systems with just a handful of engineers. Amazon's Web Services are going to make web-scale computing available to anyone. This is unimaginably powerful. This future is almost here, so we can begin thinking about it today. What kinds of things do you want to build with existing and coming Amazon Web Services?