And as of today, that service suite has increased with the addition of Web Elements, Page Speed and Tips for Hosters.

Web Elements: This service allows users to modify, copy and paste snippets of code into their websites. This is particularly useful for nonprofit organizations and other groups who do not have a full-time dedicated webmaster. Rather than updating html fields for calendars, events and testimonials, site owners simply drop in a code snippet and the site automatically updates itself from tools like Google Calendar and Google Docs. As well, instead of having to update every testimonial emailed to the group, the organization can simply paste in a conversation element or comment bar titled "Testimonials".

Page Speed: Page Speed is an open-source Firefox / Firebug extension that helps web developers optimize site performance and reduce code redundancies. Similar to Yahoo's YSlow, the tool points out all your mistakes and helps you fix them. Think of it as constructive web parenting.

Tips for Web Hosters: Tips for Web Hosters offers advice on account management, analytics, translation tools, server configurations and Ad Sense. If you're a large-scale web host and you'd like to take advantage of the "Google Services for Websites" program, apply here.

Like most of its free rivals like FotoFlexer or Picnik, Aviary can import pictures from most of the popular online photo-sharing sites, including Flickr, Facebok, and Picasa. Its features, however, go far beyond the functionality of these free online applications and the Peacock application allows you to experiment with images in ways that no other online application has allowed you to do so far.

Discount Offer

Overall, Aviary is definitely worth a try. You can sign up for a free (but limited) account here, or follow this link to claim your $55 discount if you decide to go for the paid version before November 3rd.

PayPerLive could easily become the de-facto standard for bands who want to broadcasts concerts live as a pay-per-view event. The success of this service, however, will depend greatly on the quality of the video, but also the audio stream. If our experience with watching Leo Laporte's steams on TWIT is any indication, though, Stickam should do just fine.

Still, paying users tend to be a lot more fickle than those using a free service (and those can be quite fickle already), so part of Stickam's challenge will be to provide a stable, high-quality service that doesn't go down or stutter in the middle of a concert or consultation session.

Learning Iceberg is relatively easy - Byrne says "20 minutes and you can be a programmer" - I'd argue it's a bit more than that if you haven't ever been exposed to any programming concepts, but it's far from impossible. To get you started, there's a vast and well-documented training section available from Iceberg's web site, which includes free sample applications, step-by-step guides, screenshots, diagrams, videos, and more.

Build an App in 3 Minutes

As far as what you can build with Iceberg, it's really up to you. Although the focus is on business applications, like CRM or PM tools, you can interface with anything that offers up a web service. For enterprise environments, instead of using Iceberg as a service, I.T. departments can download and use Iceberg offline, behind the firewall, to work with their in-house servers, like Windows SQL server for example.

For companies making an investment in using Iceberg, the team will even go a step further than just providing the service, but will also work with the business to help them with the process of building and customizing their apps. Several of the companies out of the 2000 or so downloads Iceberg has had to date have had this type of assistance, including companies that have moved from Zoho's CRM to their own personalized version.

Today, Iceberg is revving up to a new version: Iceberg 2.1. In this latest iteration, there are new features like a getting started wizard that makes creating applications as simple as answering simple questions about their business needs. They've also added a bug tracker, a more flexible interface, and a new embed feature that lets you "mashup" snippets from any web site or widget right into your form or application.

Google Maps Mashed Up in Form

They've also introduced what they're calling their first "killer app," a project management application for any design/development agency. With this app, users can manage projects, get intelligent estimates based on history, interact with clients, view charts, and more. Unlike with Basecamp, for example, emails are built into the system, you can automatically track and refer to all the communication about the project with ease.

Email functionality in Iceberg's PM App

If you've already written off Iceberg as just another business/enterprise app and therefore not for you, then you're missing the bigger picture. Of course, Iceberg is offering tools for business - that's where the money is - but it's also offering tools for the everyman/novice programmer. Want to build the next great Twitter app?  You can use Iceberg for that...but you can also use it to build the next Twitter, too.

If you want to try Iceberg today, you can download it from this link here for a special deal. The download includes 5 licenses, but if you choose to tell your friends about Iceberg, then you can get access to 2 more free licenses, as will your friends.

While everyone has their opinions about Facebook's future and we've given recommendations on how to make Facebook useful again, there's no denying that Facebook is a success. Facebook's integration of search, aggregation, and conversation is a winner in anyone's book.

Search
Their search feature is very useful for exploring every nook and cranny on Facebook. These days, tracking someone on Facebook is as easy as finding that same person on Myspace. Facebook's search feature is stalking at its best.

Aggregation
Facebook has been great with aggregating personal content since its all exclusive launch. With the addition of a plethora of great Facebook Apps to help out, aggregation on Facebook has only gotten sweeter. They've recently opened up the Newsfeed to include web 2.0 services Flickr, Picasca, Yelp, and Del.icio.us. While we didn't think Facebook's news feed was open enough, it was a huge leap for Facebook. Coupled with the Facebook Apps that integrate Twitter, FriendFeed, and more into the service, aggregation is becoming a strong area for Facebook.

Conversation
There's plenty of conversation about Facebook and on Facebook. The Wall application is one of Facebook's most popular features and definitely enhances the service. Really, Facebook speaks volumes in this area, so we won't bore you with a rehash.

Twitter

Our beloved Twitter has received a lot of complaints in the past week for its downtime problems. Non-Twitter users even may wonder why we put up with the service. Three words: search, aggregation, conversation.

Search
The integrated search function on the Twitter site isn't all that useful. It took over a year for the feature to be implemented and like Facebook, Twitter apps do the job better. While search is basic on Twitter, search engines like Summize, Terraminds, and Tweetscan do more than enough to solve the various problems that Twitter's official search engine doesn't. With these search tools, you can better see what people say when they tweet, track your brand on Twitter, and more. In fact, two of these engines are Twitter's most popular tools, helping to make Twitter a killer service.

Aggregation
Twitter doesn't discriminate on what you can add to the service. Pulling in feeds from other sites and pushing them to Twitter is simple. Since Twitter doesn't do this by default, you'll have to use one of the many twitter tools available such as Twitterfeed. Grab the RSS feed of any service you'd like to aggregate on Twitter, plug it into Twitterfeed and you're good to go.

Conversation
Twitter is one massive conversation in itself. It's a communication tool so of course conversation is a focal point that Twitter is most successful at. However, what's so great about the conversations is the wide variety of topics and "tweets" being sent through the service. Content may be king for blogging, but conversation is the king for Twitter.

FriendFeed

FriendFeed is this year's Twitter, which was last years innovator. While FriendFeed may be a duplicator in some ways of other services before its inception, there's no denying that it's now the top social aggregator and a service that's gaining a lot of steam.

Search
FriendFeed is the child of some of Google's best engineers. Consequently, the search engine on FriendFeed is fantastic. Users have a multitude of ways to search for things such as going through your own aggregated items for something, searching your entire network of friends or one person, and even using the search to find something within a specific service.

Aggregation
FriendFeed is a social aggregator. What can't you import to FriendFeed if not through the various integrated services then via the "imaginary friend" feature? Enough said, don't you think?

Conversation
FriendFeed has some of the best conversations on the web. Not getting enough comments on your blog? Head over to FriendFeed to find them. Members of FriendFeed congregate not only around various popular blog articles, but also Twitter messages, shared items and more. In contrast to Twitter, FriendFeed's conversations are a lot more organized and logical. Where Twitter conversations can be noisy, FriendFeed conversations would be the exact opposite: valuable.

Continuing Success

Search, aggregation, and conversation are the focal points of today's web services. There are more blogs, alternative search engines, and aggregators then ever before. These are just a handful of services that have taken off. While these may be the keys to a great beginning, in our next post we'll follow up with the keys to expanding the success of a service. Until then, we'd love to hear your thoughts on what some other keys to success are and what keys help with the longevity of a service.

Cold Calling is Dead

The reason we all hate cold calls so much is because they are very intrusive. A stranger interrupts our flow, and takes precious seconds away from our lives. But maybe even as recently as 10 years ago we did not feel it so acutely. Why? A few reasons. First, the pace of our lives was not as fast, the minutes did not feel as precious. But more importantly, today we have a much less intrusive form of solicitation - email. True we all hate spam, but an unwanted email doesn't feel like as sharp an interruption as an unsolicited phone call.

Besides being annoying, cold calling is no longer effective. People are smarter these days, and have learned to ignore upsells. A targeted email which avoids the spam box has a higher chance of getting a response than a call. With a call, the default allergic reaction is now "no." But with a brief and sincere email it could be, "hmmm, this might be interesting..." However, even cold emails do not work. To have a chance at making a sale, you need to get a warm introduction. It used to be that the business web was hidden inside of people's heads and rolodexes. Today, however, a lot of it is out there in the open - inside a digital business network called LinkedIn.

Warm Calling via LinkedIn

LinkedIn is a business network that has emerged as a substitute to the rolodex. Because it is online and self-managed, LinkedIn offers a much more robust way of maintaining your business connections and seeing what they are up to. But beyond that, LinkedIn has become an indispensable tool for business introductions.

Say you're interested in talking to Acme Co. about your new product. You log into LinkedIn and search for people who work for Acme. Then you see how you might be connected to them. Ideally connection is just one degree away, or in other words, you know someone who knows the person you are looking to connect with directly. And then you ask for an introduction.

An introduction received via LinkedIn is much warmer than a cold call, because it comes with a bit of trust. You are no longer a stranger trying to upsell things that no one needs, instead you come with a recommendation, however light, from a person that the receiver is connected to. And even if you can't find a path to connect to someone, sending a direct message via LinkedIn is better than sending a cold email. The reason is that LinkedIn implies business context, and so the person you're trying to reach likely is not going to be as surprised or angry about the unsolicited ping.

Creative Calling via Social Media

Beyond connecting on LinkedIn there are other modern means of connecting with people. Facebook message, Twitter @response, a comment on a photo or blog post, etc. These are ways of getting someone's attention that are creative, but you need to be careful when employing them because they can be unwelcome. People do not use Twitter to get unsolicited business pings, nor do they post pictures for strangers to comment on. Facebook is probably somewhat acceptable because a lot of people are mixing business contacts with friends there. But the most solid way of connecting with someone outside of LinkedIn is via their blog.

People who blog generally want to have a conversation. If you engage with someone around their blog and participate in a conversation on a topic that they are interested in, you will naturally connect with them. Particularly if your business engagement is relevant to the topic they are discussing, blog comments are likely the best way to engage. However, if you try to push the conversation off topic, the person will perceive you as disingenuous and there will be no business.

Transaction 2.0

Let's suppose you've found the right way to connect and you've got your meeting. Now you're looking at the whole sales cycle. Particularly, if you are small startup aiming to sell your product to a big company, has anything changed? Not really. You still have two fundamental hurdles - the time and the risk. Between startups and big companies expectations of how quickly the deal can get done are completely misaligned. Big companies are scared of the startup speed. Startups are frustrated with big companies' turtle pace.

Beyond the length of the sales cycle the issue that kills most transactions between startups and large firms is risk. Will this 5 person company be around tomorrow? That's a question that large companies are likely to answer with a "no" and that becomes a big problem. For this reason it doesn't make sense to buy from startups - it is too risky. However the mitigating factor is often cost - startup products are often cheaper or even free. Yet even if the technology is free and easy to remove if things don't work out, big companies are wary. They do not understand free, it scares them and perhaps rightly so.

The worst part about having a startup that sells to big firms is actually scale. The famous crossing of the chasm necessary to get big is really complicated. In the enterprise world, it means signing up many clients, keeping the pipes open, and sending out more and more products. This model is so costly and risky that venture capitalists are reluctant to shell out the money to fund it. Because of the complexity of building the enterprise business that knocks on doors a new model is emerging - web services and APIs.

Door Knocking 2.0: Web Services and APIs

How can a small start up that has no capacity to knock on doors sell to big companies? A possible answer can be via a web service or an API. The model is applicable to a whole range of services - from data plays like del.icio.us to messaging systems like Twitter to infrastructure like Amazon Web Services and semantic web services like Open Calais from Reuters. The basic model is to have a web service which is accessible via API (application programming interface). Clients sign up to use the service and have to agree to the terms in order to obtain a key. Using those keys, clients can use the service programmatically to send and get data from it.

Some examples: the del.icio.us API, allows clients to access information about specific users (if the user permits that). The Twitter API allows sending and receiving messages without using the Twitter web site. The Calais API is an example of a web service which encapsulates an algorithm. In this particular case, the algorithm takes a document and extracts semantic information from it. Unlike del.icio.us, which offers an interface to consumer data, Calais is a one shot deal algorithm. And perhaps the most important example of a web service play comes from Amazon. Taken collectively, the offered Amazon services is powerful infrastructure for building web-scale applications.

What is common between all these web services is the simple monetization strategy - pay per API call. For each call into the web service, the callee has to pay based on the amount of the resources consumed by the call. For example, Amazon has been charging for bandwidth, storage, and CPU time. The exact model does not matter as long as a fraction of a cent is charged for each call. Remarkably, this is a business that has a huge potential to scale. Each individual client is paying an affordable price, because each call into the web service is very cheap. However, collectively clients might amount to big revenue for the service provider.

What is the most attractive about this business model is that it is completely forecastable. By estimating the cost of scaling the business (mostly hardware, support and maintenance) and setting the price per web service call and the number of clients, you can determine if the business will work or not. Of course to be fair, we need to mention that just like in traditional sales, there is number of clients hidden in every equation. Two fundamental risks exist in this model - clients will not want to use the service and clients might not be able to figure out how to use it.

Still, the risks and costs of a web services based business are much less than the traditional enterprise approach. There is no need for an expensive sales force and an army of consultants to implement the solution. We are yet to see this model succeed in a major way, but because of their simplicity and straight revenue model the API based businesses are looking attractive.

Conclusion

Nothing stays constant in this world. The technology, the web and the society always evolve. Business development evolves along with everything else and lead generation has been changing along with methods of communication. Business networks like LinkedIn have replaced old rolodexes and email have made cold calling look ridiculous. Yet, there are no fundamental changes in the sales cycles and risks for startups that choose to go the traditional route of knocking on the doors of large companies.

The markets are iterating to come up with a new form of business development called web services. This new form is both cheaper and simpler - no enterprise sales force is needed to scale the business. However the question, "If we build it, will they come?" still remains unanswered. If any company can make this model work really well it is likely to be replicated and become widespread. Will web services succeed? Time will tell.

For now, please share your favorite examples and stories of business development 2.0 in the comments.

The Mechanical Turk service, which Amazon released in November of 2005, is a web service that allows companies to outsource simple, generally repetitive tasks to human workers for small sums of money. It now has 100,000 workers in 100 countries and counts corporations such as comparison search engine PriceGrabber among its users.

The service got mainstream attention when Amazon used it to help organize a virtual search for missing Microsoft researcher Jim Gray last year.

A quick survey of open assignments on the Mechanical Turk site reveal a handful that pay up to $15.00 but the vast majority paying out under $1, and many paying only a few cents. There are over 31,000 so-called Human Intelligence Tasks available on the site right now, but scanning through them by price makes it easy to imagine that collectively they're still probably not worth as much as what some of Amazon's executives can find in their couch cushions. So it's not hard to see how some people could accuse Amazon of creating a virtual sweatshop labor force.

Who Are These People?

A recent demographic survey done through Mechanical Turk sheds some surprising light on just who these "Turkers" are, however. Contrary to the pictures painted by some media outlets that Amazon has assembled a third world workforce of people willing to work for pennies, most Turkers are actually from the United States. According to the survey, 76.25% are from the US, with just over 8% from India.

Further, the vast majority of Mechanical Turk participants are under 40 years of age, and over 50% of them have bachelor's degrees. About half also make over $25,000 per year -- and a surprising percentage make over $40,000 per year.

So why participate in the Mechanical Turk program -- one which nets most people under $600 per year -- if you're well educated, already have a paying job, and there are so many other ways to make money? One answer is that people find the tasks offered on Mechanical Turk fun. The Amazon service provides people with time wasters that also pay a little money and for younger users, especially, the service offers an easy way to make a little pocket change.

A recent New York Times article relates a number of anecdotal stories about why people participate, as does an earlier post on Panos Ipeirotis's blog (Ipeirotis is responsible for the demographic survey referenced in this post). Clearly, very few people participate in Mechanical Turk solely to make money. Most people do it out of boredom, to make a little pocket change, or because they are limited in the type of work they can do due disability or some other reason.

Have you ever used Mechanical Turk to outsource a task? Have you ever participated as a worker? What was the result? Let us know in the comments.

FuseCal, a product of Public Display, a start-up technology company located in Providence, RI, is currently in alpha mode. The service works with many different types of calendars, both offline and online, including Outlook 2003/2007, Google Calendar, Apple iCal, Windows Calendar, Yahoo! Calendar, and more.

How It Works

To use the service, you just enter in a URL of any web calendar, and FuseCal will display it so that you can pick and choose which appointments you want to sync. The calendar doesn't have to be public - even a private Google Calendar URL will work.

Beneath the preview of the calendar, two events panes are displayed: "Events I Don't Want" and "Events I Want." Using the "Filter" box, you can narrow down the list by keyword or you can pick and choose events to sync manually, using the arrows.

Filtering Events in FuseCal

When you're finished filtering the calendar, you click on the "Save Events To My Calendar" button and choose the personal calendar you want to sync with. You can also choose to automatically add any new events from this feed to your calendar, not add new events, or add future events containing a particular keyword or phrase.

Setting up Syncing

FuseCal For Web Publishers

For publishers, FuseCal can also be used on your web site. For example, you may have an online calendar of your organization's events for the year which your web site visitors would be interested in syncing to their own calendar. By using the option in FuseCal for publishers, you can put an embed code on your site that allows your visitors to take your web calendar and convert it into an iCal format they can subscribe to. Each event keeps a link back to the original source URL, so your visitors can refer back to your web site at any time.

Publishers can also keep track of their calendar's usage via real-time analytics, showing subscribers, hits, times, locations and more. FuseCal even promises that publisher's calendars are human-reviewed for accuracy.

Unlike other services, like SyncMyCal, there's nothing to download and you don't have to limit yourself to only a few types of calendars. The FuseCal site also has an intuitive UI making it easy-to-use, even for those who aren't tech-savvy. I did have some occasional problems with the site's responsiveness, but that will probably be resolved as they grow. For what they are trying to accomplish, FuseCal does a great job, especially considering they have only just launched into alpha. This is definitely a startup to watch this year.

In this post we list 10 of our favorites, but there are many more of them to explore. We invite you to add your favorites in the comments, along with a note about why you like them.

First some context. If you're a last.fm user, you may not be aware that powering it is a database system known as Audioscrobbler - described on its homepage as "a massive database that tracks listening habits and calculates relationships and recommendations based on the music people listen to." Audioscrobbler started out as a plug-in that tracked a user's listening patterns on audio players such as iTunes, Winamp, Windows Media Player, etc. It was eventually integrated into last.fm and is now an important part of its recommendations system - not to mention the key to its API, which allows third party services to use the data.

At a high level, this is what Tom Coates was refering to last week at Webstock as the Web of data. The Audioscrobbler webservices page goes into more detail about what data is made available - a wide range of information, from artist data (tags, most popular albums, etc) to track data to group data and more. But for this post let's focus on 10 of the best mashups that use this data. In no particular order...

1. MusicPortl

A very slick Ajax powered search engine that delivers videos, photos, blog entries, albums and other info about a particular music artist. Last.fm data is featured, along with data from Amazon, Flickr, Ontok, MusicBrainz, YouTube and Wikipedia. What I especially like is that it features blog posts that are related to the artist, via Technorati.

2. Tokyo Stage

TokyoStage mashs up music charts, Youtube, LastFM, and LyricWiki. Its charts are updated daily. Here is an example, featuring the current number 1 in the US, 'Low' by Flo Rida Featuring T-Pain. The below screenshot shows a number of different videos that use the 'Low' music, many of them not by the artist.

3. The Pirate Bay music section

As TorrentFreak reported in December, Bit Torrent tracker site The Pirate Bay launched a new music section where artists are categorized via tagging data from last.fm. The music section is still a beta release, but you can see from the screenshot how it makes music discovery easier.

4. musicmesh

Lets you explore music derived from audioscrobbler by a graph of album covers, which are sourced from Amazon. You can watch videos from YouTube and see reviews at Amazon, Wikipedia etc. Nice search and tag exploration options too.

5. TuneGlue

A cool visualization of music artists, using Last.fm data mashed up with with Amazon data for things like album details. You can also control the "friction" and "elasticity" settings, to change up the results.

6. Last.fm normalised rankings

Awkward name, but this neat little service gives you your own Billboard-like chart of what you listen to the most. It ranks artists and albums by estimating how long you have spent listening to them. All you need to do is input your last.fm username. For what it's worth, here is my top 10 (which pretty accurately reflects my subjective favorites!):

7. Music Artist Cloud

This service generates a tag cloud of artists similar to the one you've input, based on Audioscrobbler data. I generally don't find such tag clouds to be overly useful, because music is very subjective - e.g. Bryan Adams isn't very similar to REM in my view. Still, there are some useful aggregations of YouTube video and Amazon links in this service.

8. One Hit Wonders chart

If you've ever wondered where Patrick Swayze's hit song 'She's like the wind' ranks among last.fm's user base, wonder no more... It's right up there at number 4! One hit wonders come from a suggestion box on the site's homepage. This is a fairly useless list, but it is a fun way to showcase how last.fm data can be mashed up.

9. Upcomingscrobbler

This is a more useful last.fm mashup - it's a combo of Last.fm and Yahoo-owned events website Upcoming.org. It tells you where bands are playing live in your local area. Bandsintown is a similar service.

10. PandoraFM

A frustrating app for many of us who can't access Pandora outside the US. But provided you are living in the USA, each song you listen to in Pandora is submitted to your profile on Last.fm. It claims to be "the best of both worlds" -- but what a great shame most of the world can't use it.

Other ReadWriteWeb Articles on last.fm:

• Visualizing Last.fm's Friends Network
• How Last.fm Will Create "Communities Around Content"
• Songza, Last.fm Expand Music Libraries
• CBS Now Buying Web 2.0? Last.fm for $280m
• Last.fm Sale Fallout: User Response

He starts by saying that the companies that have done well on the Web (Facebook, amazon, flickr, twitter, dopplr, etc) are much more than websites - they've broken out of the browser/page and manifesting themselves elsewhere (devices, other sites, etc). Moreover, they're all platforms and have benefited as a result; and some have made lots of money.

MOO - can print photos from services such as Flickr

The world of tomorrow? It will become "more bizarre and weird" as the Internet touches more things. He gives 3 examples:

1) A physical object responds to or visualizes data from the network.
2) Interacting with a physical object allows people to change data stored in the network.
3) A physical object acts as a sensor that writes to the web of data.

He talks about Nabaztag, a network-enabled rabbit toy. Ambient Orb, Wattson, Weather Underground ("like last.fm for weather") are other examples. He also points to AppleTV - "one of the few mainstream physical network appliances available". See last100's recent review of Apple TV.

So, when you're building a new web product it must play well with others - "it's good to design for recombination". Coates asks: what are the opportunities for your product if it can play well with others?

Weblogs and RSS are the technological background for all of this. He says this was the first step in the consumer web - where data is discoverable and explorable. Since then apps like delicious and flickr have expanded on this. So why open up your data? Coates lists 4 reasons:

1) Drive people to your service
2) People will pay for it
3) as advertising or to put yourself in the middle of an ecosystem
4) Make your service more attractive with less central development (eg flickr)

He says the main reason though is: network effects. New services can build on other services/data, and this makes the overall ecosystem more powerful. Oakland Crimestopping is an example.

Coates then talks about Fire Eagle, his latest project at Yahoo Brickhouse. Fire Eagle is described on its website as "a new way to share your location with friends or with other websites and services". It's all about geo-coding and the idea is for developers to "build all kinds of applications that respond to your location". Coates says that Fire Eagle has "APIs in two directions", so you can do geo-presence applications. He says that while the Fire Eagle site itself doesn't necessarily do much, other services can use the data and create powerful apps.

Coates next point: you can never have too much data. He follows up by saying that hierarchies can't take the weight - so we need to move to "weblike exploration". This latter point seems to complement well Peter Morville's theories about findability. Amazon is a great example, says Coates. They have moved to manifest their data outside the categories, into the pages themselves. i.e. Amazon used to have tons of tabs, but now you can explore its data from inside Amazon webpages (and in other services too). For example the "people who bought this" feature, tags for books, and Listmania. So, he says, top navigation is now just a jumping off point - there are lots of other ways to explore data nowadays.

Conclusion

As always from Tom Coates, an enlightening presentation - with many things for designers and entrepreneurs both to take away. The interesting thing for me about watching this presentation is how many real world examples Coates now has to back up his theories. When I first began following Coates' blog Plasticbag (and sadly he no longer writes long-form posts on it, just links), he was ahead of his time and - as he himself noted in this talk - only blogs and RSS were really representing the Web of data. But now we have many mainstream web companies with products that demonstrate the Web of data. And there is a lot more to come, as the Internet permeates real world objects and different devices.

Yet, even from a casual glance, it is clear that Amazon's work is very important. Since early last year, the e-commerce giant has been making forays into becoming a Web OS company. Amazon has been methodically exposing pieces of its own infrastructure as commodity web services, and in the process confusing Wall Street analysts and making thousands of startups quite happy.

Dr. Vogels has been both the architect and the evangelist of this effort. In his speech at last year's ETech conference he explained that by leveraging the Amazon Web Services Stack, web developers are finally able to focus on the core business logic of their apps and services. Hiding the enormous complexity of building a scalable web business behind a simple API, Amazon is paving the way toward a whole new web ballgame.

What Vogels talked about on his blog a few weeks back is not a public web service, but a piece of internal infrastructure, which allows Amazon to service millions of customers. The paper is a unique revelation about the inner workings of one of just a handful of Internet giants. It is also a preview of the web services to come in the next decade. In this post we take a close look at Dynamo, discuss where it fits and consider its implications.

Scalability Issues With Relational Databases

Before discussing Dynamo it is worth taking look back to understand its origins. One of the most powerful and useful technologies that has been powering the web since its early days is the relational database. Particularly, relational databases have been used a lot for retail sites where visitors are able to browse and search for products. Modern relational database are able to handle millions of products and service very large sites.

However, it is difficult to create redundancy and parallelism with relational databases, so they become a single point of failure. In particular, replication is not trivial. To understand why, consider the problem of having two database servers that need to have identical data. Having both servers for reading and writing data makes it difficult to synchronize changes. Having one master server and another slave is bad too, because the master has to take all the heat when users are writing information.

So as a relational database grows, it becomes a bottle neck and the point of failure for the entire system. As mega e-commerce sites grew over the past decade they became aware of this issue - adding more web servers does not help because it is the database that ends up being a problem.

Dynamo - A Distributed Storage System

Unlike a relational database, Dynamo is a distributed storage system. Like a relational database it is stores information to be retrieved, but it does not break the data into tables. Instead all objects are stored and looked up via a key. A simple way to think about such a system is in terms of URLs. When you navigate to the page on Amazon for the last Harry Potter book, http://www.amazon.com/gp/product/0545010225 you see a page that includes a description of the book, customer reviews, related books, and so on. To create this page, Amazon's infrastructure has to perform many database lookups, the most basic of which is to grab information about the book from its URL (or, more likely, from its ASIN - a unique code for each Amazon product, 0545010225 in this case).

In the figure above we show a concept schematic for how a distributed storage system works. The information is distributed around a ring of computers, each computer is identical. To ensure fault tolerance, in case a particular node breaks down, the data is made redundant, so each object is stored in the system multiple times.

In technical terms, Dynamo is called an eventually consistent storage system. The terminology may seem a bit odd, but as it turns out creating a distributed storage solution which is both responsive and consistent is a difficult problem. As you can tell from the diagram above, if one computer updates object A, these changes need to propagate to other machines. This is done using asynchronous communication, which is why the system is called "eventually consistent."

How Dynamo Works

The technical details in Vogels' paper are quite complex, but the way in which Dynamo works can be understood more simply. First, like Amazon S3, Dynamo offers a simple put and get interface. Each put requires the key, context and the object. The context is based on the object and is used by Dynamo for validating updates. Here is the high level description of Dynamo and a put request:

• Physical nodes are thought of as identical and organized into a ring.
• Virtual nodes are created by the system and mapped onto physical nodes, so that hardware can be swapped for maintenance and failure.
• The partitioning algorithm is one of the most complicated pieces of the system, it specifies which nodes will store a given object.
• The partitioning mechanism automatically scales as nodes enter and leave the system.
• Every object is asynchronously replicated to N nodes.
• The updates to the system occur asynchronously and may result in multiple copies of the object in the system with slightly different states.
• The discrepancies in the system are reconciled after a period of time, ensuring eventual consistency.
• Any node in the system can be issued a put or get request for any key.

So Dynamo is quite complex, but is also conceptually simple. It is inspired by the way things work in nature - based on self-organization and emergence. Each node is identical to other nodes, the nodes can come in and out of existence, and the data is automatically balanced around the ring - all of this makes Dynamo similar to an ant colony or beehive.

Finally, Dynamo's internals are implemented in Java. The choice is likely because, as we've written here, Java is an elegant programming language, which allows the appropriate level of object-orineted modeling. And yes, once again, it is fast enough!

Dynamo - The SLA In A Box

Perhaps the most stunning revelation about Dynamo is that it can be tuned using just a handful parameters to achieve different, technical goals that in turn support different business requirements. Dynamo is a storage service in the box driven by an SLA. Different applications at Amazon use different configurations of Dynamo depending on their tolerence to delays or data discrepancy. The paper lists these main uses of Dynamo:

Business logic specific reconciliation: This is a popular use case for Dynamo. Each data object is replicated across multiple nodes. The shopping cart service is a prime example of this category.
Timestamp based reconciliation: This case differs from the previous one only in the reconciliation mechanism. The service that maintains customers' session information is a good example of a service that uses this mode.
High performance read engine: While Dynamo is built to be an "always writeable" data store, a few services are tuning its quorum characteristics and using it as a high performance read engine. Services that maintain a product catalog and promotional items fit in this category.

Dynamo shows once again how disciplined and rare Amazon is at using and re-using its infrastructure. The technical management must have realized early on that the very survival of the business depended on common, bullet proof, flexible, and scalable software systems. Amazon succeeded in both implementing and spreading the infrastructure through the company. It truly earned the mandate to then leverage its internal pieces and offer them as web services.

How Does Dynamo Fit With AWS?

The short answer is that it does not, because it is not a public service. The short answer is also shortsighted because there are clear implications. First, is that since Amazon is committed to building a stack of web services, a version of Dynamo is likely to be available to the public at some time in the future.

Second, Amazon is restless in its innovation; and that applies to web services as well as it applies to its retail business. S3 has already made possible a whole new generation of startups and web services and Dynamo is likely to do the same when it comes out. And we know that more is likely to come, as even with Dynamo, the stack of web services is far from complete.

Finally, Amazon is showing openness - a highly valuable characteristic. Surely, Google and Microsoft have similar systems, but Amazon is putting them out in the open and turning its infrastructure into a business faster than its competitors. It is this openness that will allow Amazon to build trust and community around their Web Services stack. It is a powerful force, which is likely to win over developers and business people as well.

The Future - Amazon Web OS

To any computer scientist or software engineer to watch what Amazon is doing is both awesome and humbling. Taking complex theoretical concepts and algorithms, adopting them to business environments (down to the SLA!), proving that they work at the world's largest retail site, and then turning around and making them available to the rest of the world is nothing short of a tour-de-force. What Emre Sokullu called HaaS (Hardware as a Service) in his post yesterday is not only an innovative business strategy, but also a triumph of software engineering.

Amazon is on track to roll out more virtual web services, which will collectively amount to a web operating system. The building blocks that are already in place and the ones to come are going to be remixed to create new web applications that were simply not possible before. As we have written here recently, it is the libraries that make it possible to create giant systems with just a handful of engineers. Amazon's Web Services are going to make web-scale computing available to anyone. This is unimaginably powerful. This future is almost here, so we can begin thinking about it today. What kinds of things do you want to build with existing and coming Amazon Web Services?