mapreduce - ReadWriteWeb

Amazon Rents Out MapReduce Power with EC2, S3 and Hadoop

Phil Glockner — Thu, 02 Apr 2009 11:00:00 -0800

Amazon announced today that it is bridging two of its web computing services, EC2 and S3, with Hadoop, an open-source project that brings the same distributed data processing power as Google's MapReduce. In fact, it is calling the new service Amazon Elastic MapReduce. The new service will allow its EC2 customers to perform distributed MapReduce queries on enormous datasets stored in S3, paying only for the computation time they need.

]]> Hadoop has been an open-source project in the making for the last few years, inspired by Google's white paper on its version of MapReduce. The technology is an almost perfect fit with Amazon's growing web services, matching distributed CPU time with vast data storage requirements, both things that fit well with the cloud model.

The way MapReduce works is a fairly straightforward concept: You take a problem that requires working with a giant (and we're talking massive - sometimes petabytes) dataset, distribute working with the dataset over thousands of separate processes (called mapping) and then taking the thousands of results you get back and reducing those results into a single master result. For certain tasks, MapReduce can vastly improve the efficiency of these types of tasks, and adding more computing power gives you a linear improvement in speed.

Yahoo! has been using its own version of Hadoop for a while now. And even before this offering, larger Amazon Cloud Computing customers have already begun to use Hadoop in EC2. This is from Wikipedia's article on Hadoop:

As an example The New York Times used 100 Amazon EC2 instances and a Hadoop application to process 4TB of raw image TIFF data (stored in S3) into 1.1 million finished PDFs in the space of 24 hours at a computation cost of about $240 (not including bandwidth).

As Amazon says on its blog, "After a while [developers] tend to report that they begin to think in terms of the new style, and then see more and more applications for it." Which we believe means that MapReduce is the new, big hammer, and as developers start looking around, every dataset starts looking like a nail. This is good news for Amazon as it only stands to profit.

]]> Discuss

Web 2.0 and RSS services for hire

Richard MacManus — Fri, 27 May 2005 10:20:04 -0800

I'm on the look-out for more part-time writing, analysis or consulting work. I can't say too much about it here, but if you have any work opportunities for me - I'm available and I'd love to hear from you.

My skillset covers:
- Web and technical writing (including software specs)
- Social media analysis and research
- Web 2.0 consulting: including strategy, specs, product design, RSS and blog development, end-user documentation and marketing literature.

My niches are of course Web 2.0 and RSS. Feel free to email me at readwriteweb@gmail.com.

Bringing Web 2.0 to The People

Richard MacManus — Tue, 24 May 2005 12:11:19 -0800

Jim Cuene gave an excellent presentation to the Minnesota Interactive Marketing Association last week, on the subject of Web 2.0. The accompanying powerpoint presentation (3.2MB) is a nice overview of current Web 2.0 trends, delivered in a down-to-earth and businesslike way. I particularly liked Slide 3, a comparison between Web 1.0 and 2.0 entitled "Web 2.0: Evolution Towards a Read/Write Platform". Also covered in the presentation:

- Overview of common Web 2.0 apps like Flickr, del.icio.us and Wikipedia
- Introduction to blogging for a business audience
- Social Networks Connect Users into Communities of Trust (or interests)
- RSS Adoption is Small Currently, But it Could be an Attractive Tool for Niche Marketers
- Introduction to RSS and RSS Readers, using examples
- Users Who Are Using Web 2.0 Apps are Highly Engaged, Active and Attractive to Marketers
- Tactical Opportunities for Early Adopters & Marketers

Full marks to Jim for translating Web 2.0 trends and opportunities to a marketing and business-focused audience. There's also a podcast available, from Garrick Van Buren, featuring some interesting chit-chat with Jim and members of the Minnesota Interactive Marketing Association. However the audio from Jim's presentation itself was hard to pick up, so I didn't listen the whole way through. Still, nicely done by both Jim and Garrick and I hope others will follow their example and publish similar business-focused presentations about Web 2.0.