The way MapReduce works is a fairly straightforward concept: You take a problem that requires working with a giant (and we're talking massive - sometimes petabytes) dataset, distribute working with the dataset over thousands of separate processes (called mapping) and then taking the thousands of results you get back and reducing those results into a single master result. For certain tasks, MapReduce can vastly improve the efficiency of these types of tasks, and adding more computing power gives you a linear improvement in speed.

Yahoo! has been using its own version of Hadoop for a while now. And even before this offering, larger Amazon Cloud Computing customers have already begun to use Hadoop in EC2. This is from Wikipedia's article on Hadoop:

As an example The New York Times used 100 Amazon EC2 instances and a Hadoop application to process 4TB of raw image TIFF data (stored in S3) into 1.1 million finished PDFs in the space of 24 hours at a computation cost of about $240 (not including bandwidth).

As Amazon says on its blog, "After a while [developers] tend to report that they begin to think in terms of the new style, and then see more and more applications for it." Which we believe means that MapReduce is the new, big hammer, and as developers start looking around, every dataset starts looking like a nail. This is good news for Amazon as it only stands to profit.

We're talking about an annotated collection of all publicly available DNA sequences, including the Human Genome, huge amounts of chemistry data, machine readable encyclopedic entries about millions of different topics and an entire dump of Wikipedia. US Census data, data from the US Department of Transportation and more. It's all accessible by web applications in no time at all. What do you think this is going to change?

This includes data from:

• The Bureau of Transportation Statistics.


• DBPedia Knowledge Base - which "currently describes more than 2.6 million things including 213,000 people, 328,000 places, 57,000 music albums, 36,000 films, and 20,000 companies." All in handy semantic markup.


• The Freebase Data Dump - the giant collaboratively build semantic database on a wide variety of topics, data that high profile startup Metaweb has spent millions of dollars assembling.


• The entire English section of Wikipedia, dumped into a machine readable format.


• A number of large genetic and scientific databases.

We counted all the databases up and it passed 1 TB of available data. The company says that accessing this data is "trivial" for developers.

What are developers going to do with this data? We can't wait to find out. The prospect of mashing up, cross referencing and user interfacing with this amount of data is nearly unfathomable. Really. This data will be leveraged by all kinds of different web applications, for a long time.

You've read, or can imagine, the impact that the first Public Libraries had on human culture. Now imagine the opening up of not just this, but other libraries of data, so huge that economies of scale blast the project off beyond any analogy that could be drawn with our everyday experience or historical memories. It won't just be Amazon that offers up this kind of data - it will be relatively commonplace soon, we imagine.

It will be like a network of libraries - for robots. Robots that go to the library frequently, read very fast and make serious use of what they've learned.

Congratulations, Amazon, on passing 1 TB of public data made available. May all our robots of the future please live in peace.

According to Google, App Engine billing was one of the most requested features from App Engine developers. Thanks to this new pricing scheme, developers can now use App Engine to deploy larger and more popular applications on the service without hitting the ceiling of Google's free quotas. Developers will be able to set daily budgets for their apps and allocate this budget across CPU, bandwidth, storage, and email.

Some of the most popular apps that are currently using Google's App Engine are BuddyPoke, Lingospot, Best Buy's Giftag.com, and Mental Floss.

Pricing

Here is the new pricing scheme according to Google's blog post:

• $0.10 per CPU core hour. This covers the actual CPU time an application uses to process a given request, as well as that for any Datastore usage.
• $0.10 per GB bandwidth incoming, $0.12 per GB bandwidth outgoing.  This covers traffic directly to/from users, traffic between the app and any external servers accessed using the URLFetch API, and data sent via the Email API.
• $0.15 per GB of data stored by the application per month.
• $0.0001 per email recipient for emails sent by the application

Google first announced that it was planning to offer these pay-as-you-go resources to developers last May, and today's prices are at the lower end of the ranges that Google announced back then.

In general, Google's prices seem to be slightly cheaper and less complicated than Amazon's pricing schemes for using its EC2 and S3 service. It should be noted, however, that Amazon offers a far larger feature set than App Engine. App Engine only supports the Python programming language, while EC2 gives you access to a complete, remotely hosted, on-demand operating system.

Coming Soon: XMPP, Scheduled Task, Mail

Earlier this month, Google also announced a number of new features that will appear in App Engine in the next six month, including the ability to receive and process mail, support for running scheduled tasks, and support for sending and receiving XMPP messages.

Currently, you can only manage your EC2 instances through the new console, but over time, Amazon will give users the ability to mange other AWS services like its S3 storage service.

Ease of Use

The new console makes it especially easy for first-time users to set up their instances on EC2. While you can choose from a bewildering array of community-created instances (think custom Ubuntu mail servers), Amazon will present you with a sane amount of 'quick start' instances and relatively conservative security settings.

Amazon also created a great screencast that walks you through creating your first EC2 instance. While using Amazon's web services was already pretty manageable thanks to tools like Elasticfox, this new console gives users an easy way to control their instances, while also allowing new users to start using EC2 without too much hassle. If you always wanted to experiment with EC2, but were intimidated by the set-up process, now would be a good time to give it a try.