Amazon Rents Out MapReduce Power with EC2, S3 and Hadoop
Amazon announced today that it is bridging two of its web computing services, EC2 and S3, with Hadoop, an open-source project that brings the same distributed data processing power as Google's MapReduce. In fact, it is calling the new service Amazon Elastic MapReduce. The new service will allow its EC2 customers to perform distributed MapReduce queries on enormous datasets stored in S3, paying only for the computation time they need.
Hadoop has been an open-source project in the making for the last few years, inspired by Google's white paper on its version of MapReduce. The technology is an almost perfect fit with Amazon's growing web services, matching distributed CPU time with vast data storage requirements, both things that fit well with the cloud model.
The way MapReduce works is a fairly straightforward concept: You take a problem that requires working with a giant (and we're talking massive - sometimes petabytes) dataset, distribute working with the dataset over thousands of separate processes (called mapping) and then taking the thousands of results you get back and reducing those results into a single master result. For certain tasks, MapReduce can vastly improve the efficiency of these types of tasks, and adding more computing power gives you a linear improvement in speed.
Yahoo! has been using its own version of Hadoop for a while now. And even before this offering, larger Amazon Cloud Computing customers have already begun to use Hadoop in EC2. This is from Wikipedia's article on Hadoop:
As an example The New York Times used 100 Amazon EC2 instances and a Hadoop application to process 4TB of raw image TIFF data (stored in S3) into 1.1 million finished PDFs in the space of 24 hours at a computation cost of about $240 (not including bandwidth).
As Amazon says on its blog, "After a while [developers] tend to report that they begin to think in terms of the new style, and then see more and more applications for it." Which we believe means that MapReduce is the new, big hammer, and as developers start looking around, every dataset starts looking like a nail. This is good news for Amazon as it only stands to profit.
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
Does this announcement put Aster Data System www.asterdata.com/ a Sequoia backed company out of business?
I think this was move expected
This is great validation that MapReduce is going mainstream for data processing. Aster Data's vision is to bring the power of MapReduce to a whole new class of developers and mission-critical enterprise systems. Whether that's on-premise on in the the cloud is just a question of "platform".
Aster's core product is a high-performance analytic database for on-premise data warehousing. The nature of the Aster nCluster DBMS makes it very performant in the cloud - and it's available on both Amazon and AppNexus.
When would you use Aster’s In-Database MapReduce vs. a system like Hadoop? http://www.asterdata.com/blog/index.php/2009/04/02/enterprise-class-mapreduce/
I attended an AWS event in Los Angeles in September and there were many people asking for Map Reduce supports. As always Amazon is listening the the community and adding innovative services. We are very excited!
Does this announcement put Aster Data System www.asterdata.com/ a Sequoia backed company out of business?
I attended an AWS event in Los Angeles in September and there were many people asking for Map Reduce supports. As always Amazon is listening the the community and adding innovative services. We are very excited!
I agree with you Steve..
Todd DiRoberto
http://www.newsguide.us/art-entertainment/movies/Todd-DiRoberto-of-American-Satellite-Hosts-Independence-Day-Charity-Event-for-Operation-Bigs/
I attended an AWS event in Los Angeles in September and there were many people söve asking for Map Reduce supports.
While the benefit of EC2 is elasticity the benefit of colo is the ability to run your own hardware config and have dedicated support to admin your own cluster.
We see MUCH better support when we can have in-house admin vs external.
If I hire someone in-house and they don’t perform I can fire them. ;)
Further, the quality is much higher when you have a full time admin.
I think a hybrid model would work better if you can get your data into EC2 quickly. It doesn’t work for everyone of course.
We have a pretty static configuration so an EC2 migration isn’t on the roadmap until they lower their prices by 4x.