Way back when I was in high school I found myself in a high school science
class. It was your typical experience, replete with bunsen burners, saftey
goggles and a science teacher named Norbert. But one day Norbert had an
inspriation - he let the class watch a video of James Burke and his Connections
series, which described all of the happy accidents in technology through the
years, that have brought us where we are today. My favorite accidental
connection was the development of fine mist sprays for perfume bottles. While it
may have helped people smell better, the real combustion happened in your
automobile, where the fine mist sprays became fuel injection nozzles for the
modern gasoline engine.
Like a fine fragrant perfume, Amazon has a revolutionary technology sitting right under everyone's noses. Their happy accident? Building a reliable, scalable and robust ecommerce system. While I'm sure Jeff Bezos didn't envision his online company being compared to perfume sprays, the fact of the matter is, even after immense technological investment, retail needs a lot of perfume to make the margins smell nice-- even if you're online.
But in their quest to prove to the world that online retail is the wave of the future, Amazon has created not just a fine mist. They have unexpectedly created a vapor cloud - or internet cloud - that is ready for ignition. Most fortunately for Amazon, they've been able to build one of the world's most impressive, massively scalable datacenter systems. Most fortunately for you, they're willing to share it. And most fortunately for corporate programmers, you're about to be relevant again.
How valuable is a robust, secure, scalable and reliable computing platform that allows you to store and retrieve any kind of digital data from anywhere in the world at anytime of the day? While your average high school student doesn't have much use for one, technology visionaries and corporate titans have been willing to pay big money to build such platforms. Groove Networks ran through approximately $120 million of investment in order to build and convince the world that theirs was robust, secure, scalable and reliable. Microsoft, which purchased Groove in 2005, is spending upwards of $2 billion in order to do the same thing. Google's entire business model requires them to reinvest huge amounts of capital into their storage platform. Throw in Yahoo and Baidu and it's clear that such computing power is seen as a technological edge.
However a high school student can rent one of the best, most proven, storage platforms of them all -- for a little less than $.50 per month. If he or she is willing to lay off the occaissional candy bar, he/she can be on an equal web infrastructure footing with the richest technology companies in the world. If I was an investor in MSFT, I might even ask if investing $200 million in Amazon makes more sense than $2 billion on your own technology and starting from scratch. In fact, taking it one step further, I might even be worried that Amazon's examples treat .NET as just another language - alongside Java, Ruby, PHP, Python and Perl. Another $1.8 billion into .NET and Visual Studio might take care of that, especially for those prized corporate programmers.
Like anything worthwhile, it takes a bit of time to figure out the best way to exploit it. The obvious comes first: use Amazon's storage capacity and scalability to store data such as images or globs of backup data. Indeed, a simple scan of Amazon's Simple Storage Service solutions shows many backup, photo sharing and large email attachment services. Furthermore, most of these services are web-based and so their single greatest cost is usually bandwidth. Hence Amazon's marketing focus on web 2.0 apps as their clientele. This makes perfect sense, as it fits the world's current multi-tiered architecture: the simple browser on the client, the business logic on a web server and now the robust data store on the backend.
Many well known and successful companies are exploiting this today: SmugMug, 37Signals and MyBlogLog to name a few. Would you know it buy using these services? Probably not, unless you have a keen eye reading the urls flash by as elements on your web page are downloaded. Does it matter? Not one bit. Each of these services uses Amazon S3 in order to offload the work of sending images and documents off their servers and onto Amazon's. As a result, each company can focus on their distinctiveness and bring it to market as quickly as possible, without worrying so much about infrastructure. In some cases it's enough to prove the concept and sell to a large company in just a few months.
In the US there is a saying: "Only Nixon could go to China.". While Amazon has positioned itself as a proven resource for eager online entrepreneurs, they have also (accidentally?) created a solution that makes the client chic again - actually both the Client and the Server. Indeed, Amazon's Simple Storage Service takes us back to the 1980s, resurrects Client/Server architecture and provides it on such a scale that it actually works.
In the late 1980s, when C/S became a holy grail, it quickly sprang a leak at the supper table as large deployments grappled with networking protocol decisions, scaling issues, big capital investments for servers and, of course, maintenance. Amazon has moved these giant obstacles aside and resurrected something most thought dead. And this from an online department store!
While Amazon is still very much a consumer site selling anything they can display on a web page, at some point people will see the light and realize that a lot of C/S architecture implemented, deployed and wheezing along in corporations around the world can be refactored with services like Amazon S3. By providing the ultimate server, Amazon has made it possible for programmers to build corporate client software that a) actually scales and b) actually cuts costs - no more corporate servers to procure, house and maintain. A corporate programmer can create a client that can scale for the entire organization by using a good GUI tool and something only moderately more complicated than File.Open. Web application stacks of functionality might not look as appealing compared to this new reality. (Have you noticed that programming stacks have evolved like razor blades? We started with one, then two blades, then three solved the problem, until four blades really did the trick, and so on - in fact, aren't servers even called blades now?)
But while Amazon has provided the fuel, it's still up to the developers to provide the ignition; and right now we're dealing with sticks. By that I mean that Amazon isn't offering a database, nor server side scripting, interpreted languages, web server plugins or anything else to add complexity. Instead, at its core, is the world's greatest file server.
The collective moan you just heard was from everyone who actually lived through the client/server days rolling their eyes and recalling how difficult it was to merge everyone's changes into a single file on the server. That was indeed a huge problem and yes it did lead to C/S being discredited. HOWEVER, the problem was twofold: 1) tackling multi-user scalability issues for the server and 2) tackling multi-user data issues for the client. In the 1980s, personal computing was still in its infancy, and nascent applications struggled to meet the needs of a single user, much less two.
If we look at the path highly available and scalable servers have travelled the last twenty years, we see a single machine-- even a really big one-- was simply inadequate for the task-- it's just too prone to failure. Furthermore, data needs have exploded past workgroups or departments in a single location-- people across the globe may need access to specific data now. Although it will take a post in the future to adequately describe how Amazon and others are building scalable and reliable data services, they do require multiple data centers located around the world connected by fiber optics, with each data center housing thousands of redundant systems built for quick switching in case of any single point of failure. In other words, a far cry from a PC running IBM OS/2 with an Intel 386 and 32 megs of RAM locked in a closet.
But while the server side of the equation has advanced over the last twenty years, can the same be said for the client? Or, more specifically, for the data clients are producing? No. While operating systems are more powerful today and prettier to look at, the state of application data today is not so different than it was back in the 1980s-- a stream of data supporting the state of the application as last saved by a single user. In this day and age, multiple people need to work in single context to produce a deliverable. Unfortuantely up till now this has been dealt with using the checkout, checkin, 'hey, you overwrote my data!' design pattern. The server has improved, now it's time for clients to respond in kind.
The solution? How client applications read and write data needs to make the same advancements that server platforms made to make data storage reliable and delivery scalable. As application's become decoupled from the desktop, they should also decouple from the notion that a single user updates data at one time. Does only one person work ever work on a document? Does only one person ever participate in a project? Only if they wish to remain in the past.
Now that a superserver like Amazon S3 is almost a given (just a bit more reliability and maybe one more feature), applications can pull 'multi-user' files off, perform merge operations using local computing power and then place the file back in the cloud with an updated, combined view - ready for the next user to come along. Save doesn't have to mean 'write out my view of the data only' - it can also mean 'merge my view of the data with the group'. It might sound like a database, but it's really functionality that all apps working with groups, or in a group context, should have. The applications themselves will have the knowledge to merge files they recognize together - an excercise left to the developers of the next generation of client software.
There's a lot of refactoring to do, but this time the benefits are tangible. Entrepreneurs seem to be embracing Amazon web services, and the next wave can't be far behind. If Amazon manages to catch that next wave, they could be in for a great ride. And however it works out, in their quest for the ultimate online department store, Amazon might have finally solved the server side of the equation in Client/Server architecture. If they manage to attract the corporate programmers ready to build the next generation client pieces, then not only would Amazon again enjoy first mover advantage, but it would be for something worth a lot more than books. Even James Burke would have been proud of that connection.
Listed below are links to blogs that reference this entry: Amazon's Series of Fortunate Events.
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/2020
Here is a summary of the week's Web Tech action on Read/WriteWeb. Top Web News Earlier this week we reported that Yahoo's mobile social networking experiment Mixd had closed, after two months of testing. Actually it had shut down at... Read More
Comments
Subscribe to comments for this post OR Subscribe to comments for all Read/WriteWeb posts
Superb post. I think Amazon one of the most underrated tech companies out there. Who would have thought an online bookstore would emerge as one of the most innovative tech companies on the planet? S3 and EC2 extremely high potential as you state. Also, "little" things like their TiVo partnership that went live today...some very speculative thoughts in blog post linked below on how that could develop:
Posted by: gzino | March 7, 2007 8:36 PMtip of the iceberg?
well written.
Posted by: aaron | March 7, 2007 8:39 PMAmazon has been the hidden dragon, it's only a matter of time. Their current stock price is a very enticing opportunity.
Great Post.
Posted by: Michael Vu | March 7, 2007 9:35 PMThis is a terrific and important article. Although many skeptics remain, the world is increasingly moving toward a "plug and play" model, as ever more sophisticated services become accessible as low-cost subscription components in a business system. Entrepreneurs create value by figuring out how to mix and match these different low-cost capabilities to meet the needs of businesses and consumers.
Amazon has clearly recognized this ongoing evolution, and seen a money-making opportunity in helping to foster it. My hat is off to the Company.
When I wrote "Go It Alone: The Secret to Building A Successful Business! (HarperBusiness, Nov. 2004) I described this evolving business model. Now, my new blog Ventures Without Capital (http://www.VenturesWithoutCapital.com) finds the best examples of services that provide sophisticated tools, at low-cost, that enable businesses to grow by focusing on what they do best. The changes in the level of sophistication of services from the time I wrote the book to today is extraordinary! No doubt the next three years will show even more spectacular developments.
Posted by: Bruce Judson | March 7, 2007 9:49 PMAmazon is definitely on an interesting track, even more interesting than S3 is their EC2 service. With that service, they're just a couple features away from redefining the way web hosting is handled. This whole "utility computing" concept that Amazon is pushing is something that should be on every software architects radar.
Posted by: Kimbro Staken | March 7, 2007 10:10 PMThat was one of the best written articles on Amazon that I have ever read. Excellent work!
Posted by: awesomo | March 8, 2007 10:46 AM@Bruce - I think (and I believe everybody at R/WW believes) we are moving toward a 'best of breed' solution mashup, and services like Amazon's are right in there. This does two things: 1) makes it easier for the smaller players to compete and 2) makes it harder for the bigger players to prevent competition. The next couple years should be fun.
@kimbro - You know, I have to say I'm a little skeptical about 'utility computing'. I'm not an expert in that field by any means, but it just strikes me as odd that the solution is to build this 'processor clouds' when there is so much latent processing available on desktops. It is interesting, but I like the impact of virtual machines on desktops instead of servers. But that's me :)
Posted by: John Milan | March 8, 2007 11:20 AMAmazon as the 'little company that could?' I think not. They have been leaders for awhile. No reason to stop now!
Posted by: sewdough | March 8, 2007 11:29 AMAmazon has always been one of the most innovative companies on the net, even moreso than Google. Before adsense they had associates, a9 had a number of features before any other search engine, mechanical turk,"web 2.0" features before there was a web 2.0, and s3 is the kind of thing Google/Microsoft/Yahoo ought to be offering but aren't.
Posted by: Eric | March 8, 2007 11:49 AMI just don't see a mass market opportunity for amazon in this. Hardware is cheap and bandwidth is cheap. Why would any company start off relying on an Amazon's infrastructure when they can build their own and not pay any rental.
This article rehashes the fuzzy logic of the dotcom days. Remember when everyone was saying that the future of entrepreneurship was the virtual business? Where you would just repackage software and services from a number of sources on the internet and resell them. Unless you are adding value then your company has no value. In this case I think you can add value using amazon's infrastructure, but why would you take on a high variable cost base rather than an upfront fixed cost?
Posted by: eamonn | March 8, 2007 12:10 PMGreat article. I highly suggest programmers get involved in using Amazon web services. The referral compensation is wonderfully well thought out.
In college last year, I created a web service for internet enabled mobile phones that lets users check on a book's price, ratings, and reviews on Amazon.com from their mobile phone browser while they're still in the bookstore.
The service worked so well that I decided to share the service. Try it for free at www.isbnspy.com with your internet-enabled mobile phone. I use my Treo 650 to check on book prices while I'm at the local Borders or Barnes and Noble.
AWS is impressive and secure. I would definitely suggest it to any programmer wanting to get started in web services :)
Posted by: David | March 8, 2007 12:21 PM@sewdough - Leaders for a while? Definitely during the mid to late 1990s, but not in the early 2000s-- which makes what they're doing all the more impressive. Second acts are very tough in business.
@Eric - I totally agree this is the kind of thing GOOG/MSFT/YHOO should be doing-- though I really do think MSFT might be better off investing in Amazon. The inexorable march toward specialization makes implementing everything more and more difficult with each passing year.
When I ask Amazon people about Google, MSFT and YHOO joining the fray, they do tend to get a little quiet. But, if they get critical mass first, they might start talking more.
@eamonn - Hardware is cheap, and so is bandwidth? Maybe for startups, but not for scalable corporate solutions. In fact, with bandwidth, I'm not aware of any fixed costs. I know my bandwidth charges go up every time TeamDirection has a good month.
@David - You are not alone. I think Amazon is hoping you graduate (well.. *I* hope you graduate :) and build a new company with that knowledge. Then at some point, current companies will figure out how all these new companies have a better cost structure than them.
Posted by: John Milan | March 8, 2007 12:37 PMWe run our production environment in Amazon's EC2 (which itself uses S3). It's too bad that EC2 is a closed beta because I think coupled with S3, it REALLY provides Web 2.0 startups and the like with a solid, cheap, yet infinitely scalable platform.
Our site, http://www.gumiyo.com, is but one of a few sites running on Amazon's EC2 platform. Check it out when you get the chance to see what kind of performance you can expect out of Amazon's server hosting solution running alongside their storage solution.
Posted by: Dissonance | March 8, 2007 12:37 PMDamned text parser included my comma in the previous post: http://www.gumiyo.com
I fail at predicting parser behavior :p
Posted by: Dissonance | March 8, 2007 12:42 PM1. S3 can't do the basics of a webserver, like process server side includes. Not from what I can tell. In fact, it doesn't look like it can do anything a typical webserver can do, its basically just a clustered distributed version of an ftp server. Not very useful to a web developer.
2. EC2 charges 10 cents an hour to run an virtual instance of a OS so you could run a webserver that would do it. That's highway robbery. $2.40 a day, thats $70 a month, and that's just to run it. There's still an extra charge per data transfered. And no gurantee your 'instance' i.e. webserver won't die, or method to restart it.
3. When they can make a cloud of webservers to behave like this, then call me.
Posted by: Einstein | March 8, 2007 5:13 PM@Einstein
1) S3 is just a storage solution. Your application is responsible for reading and writing to it. You can allow access to the files to be directly accessible via http using REST style web services, but other than that, S3's intention is and never will be to serve as a web server. S3 is useful for a web developer that doesn't want to have to worry about a persistence layer (e.g. your own video hosting site).
2) "Each instance predictably provides the equivalent of a system with a 1.7Ghz x86 processor, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth." Transfers between outside EC2/S3 are accountable, but transfers between S3-EC2 servers are not. $70 is a paltry sum for this. Think of EC2 as a virtual computer and not just a web server. It can be a web/application/database/file/ldap/mail/whatever you want. For a web developer, you can very easily and quickly start up a cluster of servers.
3) The whole point of EC2 is to configure a base image (or more) installed with whatever software you want and to be able to run as many instances of the server as you wish. We have yet to encounter any instance failures and as long as the instance is running, you can actually reboot the operating system running in the instance (e.g. shutdown -r now). I'm sure EC2 and S3 won't appeal to everyone, but it works great for us and we've saved a ton of money using it.
Posted by: Dissonance | March 8, 2007 8:22 PMgreat post, thanks.
J
Posted by: Jesse | March 9, 2007 5:21 AMOk, the idea that Amazon has just decided to open up their internal platform for others to use is bunk. Its one of those fictional creation myths that get spread around.
Their retail platform does not use either of these technologies. This is not a "proven platform" thats been around for years... its brand new and its been showing teething pains.
The only thing Amazon already had for these services is datacenters.
Posted by: William | March 10, 2007 10:38 AMVery articulate and well written. This is new to me and very interesting.
Posted by: Randy | March 19, 2007 10:17 AM