Unless you subscribe to the unlimited plan, the only issue with Deep Dyve's new plan is that you can rent articles but not print them. This is a minor issue, however, because most users are just looking at these articles for a few facts or a bibliography and don't need them for extended periods of time. At $19.99 per month, the unlimited plan is cheaper than buying one article from a journal per month, so the price of the service won't be an issue for most of the service's target audience anyway.

Target Audience: Knowledge Workers

Depp Dyve says that its target audience is the 50 million knowledge workers in the US. This is a somewhat optimistic view. After all, how many of these knowledge workers need access to the latest articles from the Journal of Leukocyte Biology? Still, there clearly is an untapped market here, and no one but Deep Dyve is trying to exploit it.

Disruptive or Just an Extension of the Publishers' Business Model?

Deep Dyve offers users a plethora of features, including persistent searches, email, RSS alerts and the ability to bookmark articles. What is most interesting about the company, however, is this new and potentially disruptive business model. The company has indexed over 30 million articles from thousands of journals. Most of these weren't easily available to the public until now. Few users would buy an article for $30 when confronted with a journal paywall. $0.99, though, is a far more palatable price.

According to the company's CEO Bill Park, the publishers that are working with Deep Dyve believe that this new model will help them expand their market without cannibalizing their current business model, which is mostly based on selling institutional subscriptions anyway.

It will be interesting to see how users react to this new service. We think this has the potential to be a very disruptive service, especially if Deep Dyve continues to expand its database and partnerships at the current pace.

Although Netflix will try to make a streaming-only service work abroad, the company really doesn't think there's a demand for this type of offering within the U.S. In fact, when responding to a reporter's question regarding Netflix's plans for an a la carte option (there isn't one), Hastings said that while they're "open-minded to" an a la carte service that came without the DVD option, the company hasn't seen much interest in something of that nature in the States. "Everybody also wants to get DVDs," said Hastings. "All the new releases are on DVD, the vast catalog is on DVD. When there is demand, it will make sense for us to meet that demand for streaming only."

...But There is Demand for Streaming-Only

While that may be true - people do want the new releases - the demand for the physical media is arguably an artificial one created by the entertainment industry. Studios simply refuse to offer their movies and TV shows via Netflix's on-demand streaming library until they've been able to pull in a nice profit from disk sales first. This, in turn, forces consumers to not only purchase but also desire the DVD-by-mail part of the Netflix service as opposed to a separate, unbundled option of on-demand content only.

In other words, to say that the demand for streaming-only doesn't exist isn't exactly accurate. After all, Netflix reported that their streaming stats are now at an all-time high with 42% of subscribers having streamed at least 15 minutes of one TV show or movie during the last quarter. This number is up from 22% during the same period last year. Considering that Netflix's subscriber base itself has grown 28% over the past year, this figure means that the raw count of subscribers actively engaged in streaming has now more than doubled over last year. Hastings even said himself that the numbers were "a good marker of increasing streaming adoption."

Although the demand for new releases would probably have many subscribers sticking with the hybrid DVD/streaming service, by forgoing a streaming-only option it seems that the company is overlooking an opportunity to pick up a sizable group of more casual users. There are likely a number of people who would appreciate the option to pay a little less in order to to gain access to the on-demand content only - content which includes a much smaller catalog than what's available by mail. Given the company's integrations into game consoles, set-top boxes and even some TVs, there's actually no need to even own a DVD player anymore to watch Netflix movies. The content is on-demand.

Streaming media is the future, not physical disks. Hollywood knows this too, but as with the music industry, they're fighting tooth-and-nail to keep the old business model afloat for as long as possible. So far, it's working. As long as they control the method of distribution and keep it limited to physical media it will look like DVDs are what the people want. But the people really want streams. From music (Pandora, Spotify) to TV (Hulu, iPlayer) and yes, to movies via Netflix, streaming media is rapidly becoming the method of choice for many of today's consumers. The future is an on-demand world and Hollywood would do better to figure that out now than try to delay the inevitable.

Of the thousands of entries, Gavin Potter, a retired management consultant with no formal machine-learning training managed to rise to number 17 on the Netflix Prize Leaderboard. Potter writes, "The competition has trained several hundred, if not more, people how to properly implement machine learning algorithms on a real world, large scale dataset...This is, almost undoubtedly, the world's largest set of data on repeated decision making and it's ripe for analysis. The analysis may not win the competition, but it sure should provide some insights into the way that humans make decisions."

The public knowledge acquired from the process of producing these algorithms will not only affect Netflix's ability to suggest customer desires across its movie titles, but it will also form a baseline for other business systems. In addition to streaming entertainment providers, companies like Amazon and Pandora have worked hard to produce the best possible predictive technologies. If these company can tap into our unique tastes, they can suggest products and services we didn't even know we wanted. So a 10% improvement on recommendations can equate to a lucrative sales increase.

A second shorter term Netflix prize is expected in the near future. According to the New York Times' Steve Lohr, the Netflix Prize 2 will be concerned with "taste profiles" based on demographic and behavioral data.

Personalization has long been a buzzword on the Internet. With the glut of information on the Web circa 2009, personalization in this era means providing effective filters and recommendations. Ultimately personalization is about web sites and services giving you what you want, when you want it. That's the long-standing dream anyway. Let's see if the products of 2009 are fulfilling it.

Filtering the Real-Time Firehose

Personalization is often used to provide an organization layer for users on top of real-time data. As Ken Fromm put it in his primer on the Real-Time Web:

"The Internet is shifting from discrete units of websites and Web pages to discrete units of information [...] organized in ways that are relevant and personal to each individual, using data gleaned from social graphs as well as recommendation and personalization services that allow users to set their preferences."

If you use a dashboard product like TweetDeck, Seesmic or Peoplebrowsr to use Twitter, then you're able to group people, keywords and topics. This is effectively personalization at work.

Open Web: More Data About You, Better Personalization

Another aspect of personalization is the increasing prevalence of open data on the Web. A lot of companies make their data available on the Web via APIs, web services, and open data standards. And as we discussed in the first post in this series, much of that data is structured - allowing it to be inter-connected and re-used by third parties.

How does open data lead to personalization? Simply put, the more data about you and your social graph that is available to be used by applications, the better targeted the content and/or service will be to you. There are non-trivial privacy issues about this, however the personalization benefits can be significant.

There are a whole host of open data standards on the Web now. They include:

• Data portability - taking your data and friends from one site to another.
• OpenID - portable identity; single sign-on.
• OpenSocial - Google initiative for social networks, enabling developers to create widgets with one set of code; MySpace a member, Facebook isn't.
• APML - growing 'Attention' standard; Your Attention Data is all the information online about what you read, write, share and consume.

Recommendation Engines

Many consumer products on the Web aim to recommend you things that you may like. A couple of years ago, Alex Iskold outlined what he saw as the 4 main approaches to recommendations:

• Personalized recommendation - recommend things based on the individual's past behavior
• Social recommendation - recommend things based on the past behavior of similar users
• Item recommendation - recommend things based on the item itself
• A combination of the three approaches above

Amazon is probably still the best example of recommendations on the Web, but an example of something new from 2009 was Netflix launching better personalization features in March. They included new taste preferences, allowing users to (for example) choose between movies that are romantic, suspenseful, or dark. Other additions included a personalized homepage and a feature enabling users to mix and match genres.

Conclusion

Personalization has shown slow but steady progress in 2009. It hasn't been as wild a ride as Structured Data or Real-Time Web, but we consider personalization to be a key facet of the evolving Web.

ReadWriteWeb's Top 5 Web Trends of 2009:

1. Structured Data
2. The Real-Time Web
3. Personalization
4. Mobile Web & Augmented Reality
5. Internet of Things

Image credit: davepatten

If that team withstands the month long period of scrutiny that begins now, it will not only mean fame and (some) fortune for them and a big boost for Netflix - it could signal a key turning point for recommendation technology on the web.

In January of this year, we took an in-depth at the Netflix Prize, asking if 2009 could be the year that the goal would be met. In that post we discussed a New York Times profile of the contest as well, where we learned that the company's existing recommendation engine called Cinematch is credited with driving 60% of Netflix's rentals. That system is especially good at predicting "long tail" movies, older more obscure titles that are less well known but make up 70% of what Netflix customers pick. Improvements in Cinematch's effectiveness plateaued in 2006 and the move to offer a big cash prize for outside innovators has captured the imagination of thousands of engineers and their fans.

How Does it Work?

How do you judge improvements on recommendations? Netflix provides contest participants with huge piles of anonymous data about what movies certain customers rated highly, then the teams built algorithms to predict which movies other customer profiles would rate highly based on past patterns. BellKor's Pragmatic Chaos says it can now guess what people will like with a 10% improvement over Cinematch's success rate.

That gets difficult when movies like Napoleon Dynamite, which some people loved and other people hated, get thrown into the mix. It's nearly impossible to predict whether a person will like films like that.

Most of the predictive recommendation systems entered in the Netflix Prize are reported to be quite similar - so we asked in January whether it was going to take a radical breakthrough to top 10% instead of just continued iteration.

That breakthrough may have come when the four teams put their heads together, or it may have been an iterative victory. Time and science will tell.

Some people believe that recommendation as a technology has the potential to be even bigger than search. In our favorite article on the subject, written eight-teen months ago now, Dr. Rick Hangartner, Chief Scientist at recommendation engine Strands, puts it like this:

In the near term, search engines will increasingly incorporate simple recommender technologies to handle approximate queries (e.g. "You asked for this, and based on similar queries/behavior by others, you might be looking for this."). But in the long term, the recommender industry will be larger, and recommender technologies will be more pervasive than the search industry and search technology as we know it. [Because there will be recommendation going on all over the web.]

New Features: Taste Preferences, Personalized Homepages, Mix and Match Genres

Movie recommendations on Netflix, which are currently mostly based on your movie ratings, are one of the service's best features, and judging from what we have seen so far, the new taste preferences, which allow you to choose between movies that are romantic, suspenseful, or dark, for example, will make this experience only better.

The mix and match feature, too, will allow users to create a more personalized experience on the site, which is clearly the focus of today's update.

Netflix is rolling out these new updates to its over 10 million subscribers slowly, but Todd Yellin, Neflix's Director of Product Management, expects that all members will see them on their homepages within the next week.

With Queued, you can quickly access and modify your Netflix queue from your desktop, search for movies to add to your queue, rate movies, and you can even use the app to launch and view Instant Watch movies.

Since Adobe AIR lets the app run in the background, you can leave it running until you need it and when you return, there's no need to launch a browser and sign into Netflix - it's all right there for you. The app also alerts you when Netflix ships one of your movies so you know what's coming. And with AIR's offline capabilities, Queued lets you interact with it even when you have no internet connection. When the connection returns, your data will be automatically synced back to Netflix.

The Technical Details

On the Dojo side, the app uses a single HTML file for the main window, dAIR for Dojo/AIR integration, dijit for layout, unobtrusive behavior implementation using dojo.behavior, dojox.dtl for most widget templating, drag and drop for queue re-ordering, various animations for polish, and Dojo's build system.

On the AIR side, the app implements some of Adobe AIR's newest features including a local database, encrypted local storage, view source capability, automatic updates, and offline capability.

Go Get It!

The source code is available on Google code and the app itself is available for download from SitePen's web site, as they were the creators of the software.

We had the opportunity to speak with Satnam Alag, author of the recently published Collective Intelligence in Action, about what makes for a good recommendation system, where the technology is heading, and why Netflix is finding it so hard to improve its own system.

ReadWriteWeb: In our recent post about Netflix, we identified four main approaches to recommendations: Personalized recommendation: based on prior behavior of the user; Social recommendation: based on prior behavior of similar users; Item recommendation: based on the item itself; And a combination of all three. Do you agree with the four approaches we laid out in our article?

Satnam: Those four categories are pretty comprehensive. I present an alternate classification of recommendation systems in my book. I lay out two fundamental approaches. The first approach, item-based analysis, determines items that are related to a particular item. When a user likes a particular item, related ones are recommended. The second approach, user-based analysis, first determines users who are similar to that user.

Further, there are two main approaches to finding similar items and similar users. For the first, content-based analysis, content associated with the item, especially text, is used to compute similarity. In the second, the collaborative approach, actions such as ratings, bookmarking, and so forth are used to find similar items. For the second, user-based analysis, a number of approaches have been taken, including ones based on profile information, user actions, and lists of the user's friends or contacts. Of course, you can combine any these item/user and content/collaborative approaches to build a recommendation system.

The dimensions of the particular item and user space are helpful in deciding whether to use an item-based or user-based approach. Typically, an item-based approach is used to bootstrap one's application when the number of users is small. As the user base grows, the item-based approach is augmented by a user-based approach.

ReadWriteWeb: Other than Amazon and Netflix, which Internet companies have most impressed you in their implementation of recommendation systems?

Satnam: Other than Amazon and Nextflix, Google News' personalization is my personal favorite. Google News is a good example of building a scalable recommendation system for a large number of users (several million unique visitors per month) and a large number of items (several million new stories every two months), with constant item churn. This is different from Amazon's, whose rate of item churn is much lower. Google decided to use collaborative filtering for its recommendation system mainly because of its access to the data of its large user base and because this same approach could be applied to other applications, countries, and languages. A content-based recommendation system perhaps could have worked just as well, but may have required language- or location-specific tweaking. Google also wanted to leverage the same collaborative filtering technology to be able to recommend images, videos, and music, for which it's more difficult to analyze the underlying content.

Among start-ups, my personal favorite is the one we are developing at my current company, NextBio. It's not available yet but should be next month. The key point about this particular recommendation engine is its strong use of an ontology, similar in concept to tags, to develop a common vocabulary for items and users. The system then makes use of profile information and user interactions, both short- and long-term, to provide recommendations. The system leverages both item- and user-based approaches.

ReadWriteWeb: What commercial opportunities do you forsee with recommendation systems over the next few years?

Satnam: A good personalized recommendation system can mean the difference between a successful and a failed website. Given that most applications now invite users to interact and to leverage user-generated content, new content is being generated at a phenomenal rate. Showing the right content to the right user at the right time is key to creating a sticky application. I would be surprised if most successful websites did not leverage recommendation systems to provide personalized experiences to their users.

ReadWriteWeb: Your book includes a discussion of collaborative filtering. Can you tell us a bit about how this fits into the overall picture of recommendation systems?

Satnam: In recent years, an increasing amount of user interaction has provided applications with a large amount of information that can be converted into intelligence. This interaction may be in the form of ratings, blog entries, item tagging, user connections, or shared items of interest. This has led to the problem of information overload. What we need is a system that can recommend items based on the user's interests and interactions. This is where personalization and recommendation engines come in.

In my book, I take a holistic view of adding intelligence to one's application, a recommendation engine being one way to do it. The book focuses on both content-based and collaborative approaches to building recommendation systems. It focuses on capturing relevant information about the user, information from both within and outside one's application, and converting it into recommendations. One of the things you mentioned in your write-up on recommendation systems is that you would like to apply such a system to your website to recommend things to users. Someone reading my book should be able to create such a system using the techniques I demonstrate.

Next Page: Satnam's thoughts on the Netflix Prize and whether the 10% mark will ever be reached.

ReadWriteWeb: Netflix is offering $1 million to the team that can improve its recommendation algorithm by 10%. It's been over 2 years now, with the leading company at 9.63%. There is some skepticism, though, that 10% will be reached anytime soon, because now the contestants are making only incremental progress. Do you expect the 10% mark to be reached soon?

Satnam: Netflix's recommendation engine, Cinematch, uses an item-to-item algorithm (similar to Amazon's) with a number of heuristics. Given that Netflix' recommendation system has been very successful in the real world, it is pretty impressive that teams have been able to improve on it by as much as 9.63%. Of course, the Netflix competition doesn't take into account speed of implementation or the scalability of the approach. It simply focuses on the quality of recommendations in terms of closing the gap between user rating and predicted rating. So, it isn't clear whether Netflix will be able to leverage all of the innovation coming out of this competition. Also, the Netflix data doesn't contain much information to allow for a content-based approach; it's for this reason that teams are focusing on collaborative-based techniques.

The challenges to reaching the 10% mark are:

Skewed data: The data set for the competition consists of more than 100 million anonymous movie ratings, using a scale of one to five stars, made by 480,000 users for 17,770 movies. Note that the user-item data set for this problem is sparsely populated, with nearly 99% of user-item entries being zero. The distribution of movies per user is skewed. The median number of ratings per user is 93. About 10% of users rated 16 or fewer movies, while 25% of users rated 36 or fewer. Two users rated as many as 17,000 movies. Similarly, the ratings per movie are also skewed: almost half the user base rated one popular movie (Miss Congeniality); about 25% of movies had 190 or fewer ratings; and a handful of movies were rated fewer than 10 times.

The approach: The winning team, BellKor, spent more than 2,000 combined hours poring over data to find the winning solution. The winning solution was a linear combination of 107 sets of predictions. Many of the algorithms involved either the nearest-neighbor method (k-NN) or latent factor models, such as SVD/factorization and Restricted Boltzmann Machines (RBMs).

The winning solution uses k-NN to predict the rating for a user, using both the Pearson-r correlation and cosine methods to compute the similarities, with corrections to remove item-specific and user-specific biases. Latent semantic models are also widely used in the winning solution.

The BellKor team found it important to use a variety of models that compensated for each other's shortcomings. No one model alone could have gotten the BellKor team to the top of the competition. The combined set of models achieved an improvement of 8.43% over Cinematch, while the best model -- a hybrid of k-NN applied to output from RBMs -- improved the result by 6.43%. The biggest improvement by LSI methods was 5.1%, with the best pure k-NN model scoring below that. (K for the k-NN methods was in the range of 20 to 50.) The BellKor team also applied a number of heuristics to further improve the results.

The BellKor team demonstrates a number of guidelines for building a winning solution to this kind of competition:

• Combining complementary models helps improve the overall solution. Note that a linear combination of three models, one each for k-NN, LSI, and RBM, would have yielded fairly good results, an improvement of 7.58%.
• A principled approach is needed to optimize the solution.
• The key to winning is building models that can accurately predict when there is sufficient data, without over-applying in the absence of adequate data.

The final solution will be along the same lines, combining multiple models with heuristics. Contestants will probably reach the magic 10% mark in the next year or two.

ReadWriteWeb: Some people think the 10% mark can't be reached with algorithms alone, but that the "human" element is required. For example, ClerkDogs is a service that hires actual former video-store clerks to "create a database that is much richer and deeper than the collaborative filtering engines." It's a similar approach to that of Pandora, which has 50 employees who listen to and tag songs. How far do you think algorithms can go in making recommendations?

Satnam: Recommendation systems are not perfect. A number of elements go into making successful ones, including approach, the speed of computing results, heuristics, the exploration and exploitation of coefficients, and so on. But it has been shown in the real world that the more personalized you can make recommendations, the higher the click-through rate, the stickier the application, and the lower the bounce rate.

Using humans to form a rich database for recommendations may work for small applications, but it would probably be too expensive to scale. I don't see them competing against each other, human versus machine. Even with human/expert recommendations, one first needs to find a human/expert with tastes similar to those of the user, especially if you want to go after the long tail.

One of Netflix's great features is the ability to watch some of the titles it offers instantly - either on your computer, via your Xbox, or by using another external device, like the Roku player. Unfortunately, Netflix's entire catalog isn't available for streaming - only select titles marked "Watch Instantly." For subscribers who make heavy use of this feature, finding movies and TV shows to stream means tiresome browsing through the Netflix web site. But now there's a better way.

The InstantWatcher web site is Netflix mashup that filters for the "Watch Instantly" titles using the relatively new Netflix API which launched in September of last year. On the main page, the titles are grouped into genres like "Drama," "Sci-Fi," "Romance," "Comedy," etc. and within each section they are then further categorized for easy browsing.

Other Ways to Browse

Using the links at the top, you can browse by other methods, too. Options here include browsing "New," "Expiring," "Random," "People," "Best," and "Worst" titles. Who would want to find the worst movies? Well, we suppose it is funny to see how Netflix rounded out their 15,000+ titles in the "Instant Watch" catalog with a huge collection of D-list (and quite frankly, really terrible) films.

Of course, you don't have to just browse through the titles. If you have a particular movie or TV show in mind, you can just do a search for it using the box provided at the top of the screen.

Interacting with Your Queue

When you find a movie or TV show you like, you can click the "Play" button to add it to your Instant Queue. That feature makes InstantWatcher extremely useful, although we would like to see the ability to delete items from the queue as well. Perhaps in a later version they will add that functionality.

Despite that minor drawback, the Netflix InstantWatcher web site is still a great example of the amazing things which can be built when a company offers up an API for use by third-party developers. ProgrammableWeb dubbed it one of their "best new mashups" and we definitely agree. The InstantWatcher is a must for Netflix subscribers everywhere that streaming is offered.

Before we check in with the Netflix Prize, let's refresh our knowledge of recommendations technology as it pertains to the Web. Basically the idea is that given a set of ratings for a particular user, along with those of the whole user base, a recommendation system will come up with new items that the user may like. Personalization is the driving force behind this, because the more new things a retailer or service can offer a user, the more chance the user will buy / like them.

In his influential post of 2 years ago, The Art, Science and Business of Recommendation Engines, Alex Iskold suggested 4 approaches to recommendations:

• Personalized recommendation - recommend things based on the individual's past behavior
• Social recommendation - recommend things based on the past behavior of similar users
• Item recommendation - recommend things based on the item itself
• A combination of the three approaches above

The two Internet companies that have been most prominent in using recommendations are Amazon.com and Netflix. Others, such as Google, have used it as well - but more as a background enabling technology.

Netflix Prize

The Netflix Prize is a competition that Netflix - the U.S. online movie rental service - began on October 2, 2006. Its aim is to "substantially improve the accuracy of predictions about how much someone is going to love a movie based on their movie preferences." A prize of $1,0000,000 was put up by Netflix for a third party to come up with a collaborative filtering algorithm that will improve Netflix's own recommendations algorithm (called Cinematch) by a baseline of 10%. The contest has been going for over 2 years now, with no grand prize winner yet. However the latest leaderboard shows that a group called BellKor in BigChaos is closing in on the magical 10% - as of writing they are at 9.63. There are currently 7 competitors who have gone over 9%, the second best being PragmaticTheory with 9.46%.

BellKor in BigChaos is a partnership between a group of current and ex AT&T researchers (two of them still working at AT&T Labs in New Jersey) and a company called Commendo Research from Austria. They were the recipient's of Netflix's 2008 Progress Prize, with a 9.44% improvement over Netflix's Cinematch algorithm. Netflix is awarding a $50,000 progress prize every year until the 10% goal is met.

The New York Times had an extensive profile of the Netflix Prize in November. The piece notes that Netflix's current algorithm, Cinematch, was introduced in 2000 and has since gone on to be a driver for 60% of Netflix's rentals. What's more, it's also a boon for The Long Tail, because as NYT stated "it also often steers a customer's attention away from big-grossing hits toward smaller, independent movies." 70% of what Netflix customers order are from the long tail - "older movies or small, independent ones." In 2006, Netflix noted that Cinematch's improving performance had plateaued. So it released data for third parties to try and come up with improvements to Netflix's own recommendation engine. As of November 2008, the data was made up of 17,770 movies with ratings by 480,189 users.

Netflix is also busy on other fronts to tap into 'the wisdom of the crowds' - in late September 2008 it released its much anticipated API, available at developer.netflix.com. An example of the type of application this may encourage is Feedflix, a third party app that we profiled last April. It offers a variety of useful data that may help Netflix users select better movies.

Will the 10% Mark be Reached in 2009?

It's difficult to say whether the $1M Netflix Prize will be finally won in 2009. On the positive side, the current leader is only 0.37% away from claiming the prize. The NYT article suggested that the top 10 on the leaderboard all use very similar mathematical theories ("singular value decomposition" being the main one) and that differences between the teams are merely "tweaks". There's a sense though that to reach the magical 10% mark will require a breakthrough, rather than continued incremental improvements. The problem appears to be eccentric movies, the type that people either love or hate - such as Napoleon Dynamite. According to NYT, "a small group of mainly independent movies represents more than half of the remaining errors in the way of winning the prize".

Some people think that the 10% ceiling will not be reached using algorithms. ClerkDogs is a service that we profiled in December and its approach is to hire real-life former video store clerks to "create a database that is much richer and deeper than the collaborative filtering engines." In other words, it's the opposite principle to what Netflix is trying to do with computer algorithms. Founder Stuart Skorman thinks that the Netflix algorithmic approach to matchmaking has reached a ceiling; and that the only thing left to do is bring humans into the equation. He says it's a similar approach to Pandora, which has 50 employees who listen to songs and tag them. Skorman knows a thing or two about the online movie rental industry, having founded Reel.com in the mid-90s and sold it 3 years later for $100 million to Hollywood Entertainment.

We hope the prize is claimed this year, as a 10% increase in recommendation effectiveness on Netflix is a big improvement that will benefit consumers. But we also think the human element that ClerkDogs is advocating will be an essential piece of the puzzle going forward - expert human content is always the most valuable, although it generally costs more too. Perhaps Netflix will end up buying ClerkDogs? That would be an interesting mashup!

ReadWriteWeb Resources for Recommendation Technologies

We will be profiling other recommendation companies in upcoming posts. We also invite you to explore using our custom ReadWriteWeb Resources:

• RWW Recommendation Industry Feed Favorites OPML file (save link)
• RWW Recommendation Industry Feeds - Best of Feed (copy and paste to your reader)
• Click to preview the above feeds before subscribing (pop-up window)
• RWW Recommendation Site Search (Visit and Bookmark)

There was lots of activity in the digital lifestyle space in 2008, with new devices, services, and platforms being launched and some of our favorites from last year getting significant updates. One notable trend throughout the year was the way these products and services began to converge; not in the sense that they were becoming all-in-one devices, although some of that was happening, but rather by hardware, services, and content playing together nicely, often through open standards and platforms, with the Internet acting as a conduit. On that note, here are our picks of the 10 best digital lifestyle products of 2008.

• Top 10 Semantic Web Products of 2008
• Top 10 International Products of 2008
• Top 10 Consumer Web Apps of 2008
• Top 10 RSS and Syndication Products of 2008
• Top 10 Mobile Web Products of 2008
• Top 10 Enterprise Web Products of 2008
• Top 10 Real World Web Apps of 2008

1. The App Store

The real upgrade to the iPhone this year wasn't the iPhone 3G but the accompanying App Store. Launched just five months ago, the store now offers over 10,000 third-party apps, and Apple has seen over 300 million downloads. Part of that success can be attributed to the way in which the iPhone as a platform has galvanized developers; a second major factor is the simplicity of the App Store itself. As a result, lots of our other favorite digital lifestyle-related products and services wound up on the iPhone and iPod Touch, such as Pandora and Last.fm (digital music), Joost (Internet TV), Facebook, MySpace, and Twitter (social web), as well as location-based services, games, remotes (VLC Player and Sonos), and much, much more.

See also: The real surprise of the App Store isn't number of downloads or revenue

2. Netflix

When Netflix starting talking up plans to deliver its online streaming service, Watch Instantly, to "Internet-connected high-definition DVD players, Internet-connected game consoles, and dedicated Internet set-top boxes," we were a little skeptical, especially of the time frame. However, the company really delivered in 2008: Netflix streaming is now available on TiVo, the XBox 360, Internet-connected DVD players from LG and Samsung, along with the Roku Netflix Player set-top box.

3. Android

Our initial review of the first Google phone, T-Mobile's G1, was mixed, but the Android OS had us pretty excited. "Without a doubt, the Android operating system is spectacular," last100's Daniel Langendorf wrote at the time. "It's fast, with little or no lag time. It's responsive, fun to use, and full of promise." A few months on and we're still impressed. In particular, Android's mobile web browser is the best post-iPhone one yet. And likewise, the Android Market does a great job of copying the iPhone's App Store. Of course, the best thing about Android is that it's open source; as a result, we'll see it powering numerous new smartphones next year, along with other hardware, such as set-top boxes, MIDs, and GPS devices.

4. Nokia E71

In our extensive review, we described Nokia's E71 as our favorite smartphone yet. So, admittedly, this one is a very personal choice. The E71 is roughly the same size as the iPhone but has a completely different form-factor, omitting touch for a more traditional user interface and with enough room to pack in a compact but very usable QWERTY keyboard. Other pluses are the device's overall responsiveness, bundled applications, and a number of welcome improvements to the S60 line's user interface, along with decent web browsing and media playback, superb call quality, and extremely good battery life.

5. Hulu

Although online video site Hulu was available in private beta in 2007, it didn't launch publicly until March of this year. Our initial verdict was mixed, but since then the Fox and NBC joint venture has become the third biggest video destination in the U.S., according to Nielsen. Perhaps a testament to that success, a number of device makers have released set-top boxes marketed on their ability to put Hulu content on the TV, such as ZeeVee's recently announced PC-to-TV solution, the ZvBox, and the Neuros LINK. Now, if only Hulu would release an iPhone app or, like Netflix, form official partnerships with consumer electronics companies.

6. BBC iPlayer

Hulu could certainly learn a thing or two from the iPlayer, the BBC's TV catch-up service (UK only). Since its controversial Windows launch, when the public broadcaster was accused of getting too close to Microsoft, the iPlayer has added streaming for the Mac and Linux, a version for the iPhone and iPod Touch, numerous other portable media players, and support for the latest phones running Windows Mobile. There's also an iPlayer application for select Nokia phones and a browser-based version optimized for the PlayStation 3 and Nintendo Wii.

7. PlayStation 3

Sony's PlayStation 3 wasn't launched in 2008, but it certainly came of age this year. The company has always pitched the PS3 as a device that goes far beyond gaming. Instead, like Microsoft's XBox 360, it's designed to be a trojan horse in the living room, delivering a range of non-gaming content and services through the television. On that front, Sony made significant progress in 2008 by winning the next-generation format war with Blu-ray, adding DVR functionality in the UK with PlayTV, launching a video download store in the U.S., adding support for DivX video, and, finally, rolling out its own virtual world called Home.

8. Songbird

After being in development for two years, the open-source desktop music player Songbird reached its 1.0 release this month. What sets Songbird apart from the likes of iTunes is the array of available plug-ins that extend the app's functionality. For example, mashTape, one of six default add-ons, let's you delve into artist info, discography, links, and news and scroll through Flickr photos and YouTube videos. Other add-on services that ship with the player out of the box are Last.fm, Concerts, and SHOUTcast radio. With these installed, you can sync your tracks to Last.fm's online service, check out upcoming concerts in the area, and stream music over the Internet using the player. As of publication, there are over 70 plug-ins available for Songbird.

See also: ReadWriteWeb's full Songbird review.

9. Wii Fit

Nintendo has long contended that "everyone's a gamer," and now the console giant wants everyone to get fit. Announced last year but released in 2008, the Wii Fit aims to improve the health of family members through the kind of active play first seen in Wii Sports. The "game" comes with a balance board that assists with aerobic, toning, and balancing activities. A neat feature is that household members can review each other's progress on a new Wii channel.

10. The Netbook

This isn't an individual product but a whole new product category that has really taken off in 2008. Initially targeted to the education market and those wanting a third machine, netbooks are resonating with a much broader market -- and not just because of their lower price point compared to more traditional, higher spec'ed sub-notebooks. Despite years of industry propaganda, consumers are wising up to the fact that they don't have to step on the processor upgrade treadmill. Instead, in an age when more and more of our applications and data reside in the cloud (on remote servers, rather than local computers), a machine with Internet connectivity and powerful enough to run a modern web browser (a netbook, in other words) is often all we need.

Nokia Reveals iPhone Competitor And Goes to Battle With iTunes

At an analyst and media event in London this week, Nokia unveiled their company's first touch-screen phone, the Nokia 5800 XpressMusic, otherwise known as the Nokia "Tube," a device designed to compete directly with Apple's iPhone. Along with the phone, Nokia also detailed plans for their new "Comes With Music" service, a 12-month subscription service which offers unlimited downloads. There's no charge to download the individual tracks because the cost for the music is bundled into the cost of the phone.

Netflix API Launches - Here's What it Will and Won't Include

The much-awaited Application Programming Interface (API) for movie site Netflix launched this week. It looks pretty good, but there are some major limitations, too. Millions of people love movies via Netflix, making this API an opportunity for all kinds of developers to add well-known value to any other application.

See also: Evernote Hits a Homerun With API, Data Portability

Google Blogsearch Relaunches as Techmeme Killer, Across 11 Categories

In its first major upgrade ever, Google Blogsearch relaunched and looks radically different. Instead of the blank page look of Google.com, Blogsearch now looks like Google News (but uglier) - with the hottest topics from the blogosphere aggregated on the front page. Readers can drill down in 11 different categories, from technology, business, sports and entertainment. Google says you can use Blogsearch to see what the world is talking about.

RWW Predictions: Will eBay Sell StumbleUpon?

Last week rumors were swirling that eBay was looking to sell StumbleUpon. eBay purchased StumbleUpon in early 2007 for a bargain price of $75 million. We've still yet to have these rumors confirmed, but what if eBay were to actually sell StumbleUpon? We ran a prediction challenge this week asking whether eBay will sell the service by the end of this year and if so, the price tag that it might fetch. Here are the results:

Poll: Adobe Confirms Flash For iPhone - Do You Care?

At the Flash on the Beach 08 conference being held in Brighton, England, Adobe's Senior Director of Engineering, Paul Betlem, confirmed that a Flash Player is in development for the iPhone. The information was provided in answer to a direct question from an audience member during the Town Hall meeting sessions held during the conference. Also check out our poll on the topic:

SEE MORE WEB PRODUCTS COVERAGE IN OUR PRODUCTS CATEGORY

A Word from Our Sponsors

We'd like to thank ReadWriteWeb's sponsors, without whom we couldn't bring you all these stories every week!

• FreshBooks offers professional online invoicing.
• SocialText provides an enterprise wiki platform.
• WildApricot is Membership Management Software.
• E.Factor is an online community for entrepreneurs.
• Strands provides real-time recommendations of products.
• Calais brings semantic functionality into your site.
• The Web 3.0 Conference is on October 16 - 17 in Santa Clara.
• Direct Media Exchange is a solution for managing ad networks.
• Rackspace provides dedicated server hosting.
• The Defrag conference is on November 3-4 in Denver.
• Quintura is a visual-based search engine.
• Eurekster is a custom social search portal.
• MediaTemple provides hosting for RWW.
• SixApart provides our publishing software MT4.

Web Trends

Where Are All The RDF-based Semantic Web Apps?

RDF is the cornerstone of The Semantic Web, yet there still very few commercial RDF apps.

In the latest issue of Nodalities, a magazine about the Semantic Web by UK company Talis, there is an article by Talis CTO Ian Davis about the state of Semantic Web applications. Davis says that we're still in "Generation Zero" of the Semantic Web, because there are relatively few compelling apps. Specifically he notes that "there are still only a handful of applications that incorporate RDF at their heart and none of these are using the full potential of the Semantic Web." RDF is the Semantic Web's equivalent of the Web's HTML - its chief characteristic is the ability to ascribe meaning to data. We investigate...

See also: Swirrl: Newly Launched Semantic Web Wiki

Web 2.0 Gritty Entrepreneurs

When the going gets tough, the tough get going. Times are now tougher. Which makes most people head home. The half-hearted entrepreneurs, the wannabes who thought it was going to be easy, the folks with connections to VCs who could get a $5m Series A for a copycat app. Who will be left? The gritty entrepreneur of the old school who knows that it is really, really tough to build a great company. At ReadWriteWeb we celebrate these gritty entrepreneurs and in a series that kicked off this week we will be writing about them - and for them.

See also: Gritty Entrepreneurs: Jigsaw, a Profitable Web 2.0 Venture

Interview With Last.fm Founder Richard Jones

This week we interviewed one of the founders of online music service last.fm, Richard "Mr Scrobble" Jones. We wanted to find out last.fm's reaction to the launch of MySpace Music and the rise of Imeem, discuss business models in online music, and find out what's new at last.fm. We ran the interview in 3 parts, over 3 days. Part 1 discusses the increasing competition in online music this year. See also Part 2, on business models and Part 3, on design and features.

5 Great Science Books to Expand Your Mind

From the dynamics of social networks to market bubbles, science has a lot to say about the world of technology.

One of the great discoveries of modern science was the realization of how interconnected the world is. The deterministic, Newtonian view of a clockwork Universe was replaced by the much more dynamic, uncertain and entangled world of Quantum Mechanics. The new world is the one where Godel forever cut hopes for completeness in mathematics and Turing showed that computation, like the future, is fundamentally unpredictable. Despite these unexpected setbacks, modern science is wonderful, powerful and thought provoking - and relevant to technologists.

SEE MORE WEB TRENDS COVERAGE IN OUR TRENDS CATEGORY

RWW Enterprise Channel

Mumboe Uses Semantics To Pull Key Data From Contracts

Mumboe isn't just another enterprise collaboration suite. Instead, they focus on doing one thing and doing it well: making business agreements searchable. That's a very unique need they fill, which is why is why they already have 3000 customers using their free Express solution after only having launched earlier this spring.

To compete with the handful of other vendors in this narrow space, Mumboe has now added a new feature called On-Demand Contract Intelligence, which takes advantage of the service's semantic processing engine to deliver something the others don't: automatic extraction of data.

Email us if you're interested in writing for ReadWriteWeb's Enterprise Channel.

SEE MORE ENTERPRISE COVERAGE IN OUR ENTERPRISE CHANNEL

That's a wrap for another week! Enjoy your weekend everyone.

Millions of people love movies via Netflix, making this API an opportunity for all kinds of developers to add well-known value to any other application.

Presumably this does not mean that 3rd party applications will be able to pull in the streaming content available on the Netflix site, but rather that they'll be able to make user data portable for offering personalized content on their applications based on a user's Netflix activities. Users will still have to visit the Netflix site itself or to one of the big integration partners like LG or Xbox in order to watch streams. Last week Netflix cut deals with the Disney Channel and CBS to put nearly 100 of their shows on the Netflix site. You vibrating hamster Facebook app will not be able to show your users their favorite Netflix video inside the hamster, though, you just get to interact with their list of favorites content as data.

Update: Now that the API is live, we see that our assumption here was wrong. As reader Dave Jeyes points out in comments, the documentation includes code for a play button and a media player. Cool!

It also appears that this is a read-only API, meaning that movies cannot be requested or other account information changed, from inside of 3rd party applications. We presume there will be affiliate links made available so that users can click through and developers can make some profit.

Update: It turns out our hunch was wrong about read-only. In an email response to the question, the company says: "Write capabilities: yes. We wanted to enable full movie queue management, so there are write capabilities in that a user can add movies to their queue, reorder their queue, and remove movies. Ratings can also be written, i.e. a user can rate a movie using the API."

Speaking of developer profit, commercial use of the API will be accepted. Netflix says, for example, that developers can sell an app in the iPhone app store that uses the Netflix API. That's great, there are far too many commercially desirable APIs around the web for which commercial use is prohibited.

The API includes access to data via REST API, a Javascript API, and ATOM feeds. No JSON, which we suspect will disappoint some developers.

User authentication will occur using OAuth, the open standard we and others have been cheering for and the protocol now used for all the Google Data APIs.

So the good news is that Netflix is using standards based authentication, making it very easy to develop against, is allowing commercial use and is finally launching the interface. The bad news appears to be that it's user-data only and appears to be read only. We'll update this post if we get any more details from the company before launch tomorrow. Update:Now that we've seen more information about the API, it appears that there is no bad news. This is great stuff.

The API will be available at http://developer.netflix.com by self sign-up tomorrow. That site is currently password protected.

Boston Globe writer Scott Kirsner tested the service last week and saw even faster turn around - his documents were available on the Pixily website in one day and returned to him in paper form in two days after sending them. That's pretty awesome.

Pixily offers subscription plans from $5 to $60 per month, for your first 50 to 200 pages mailed in and with 1,000 to 12,000 pages of storage. All stored documents are made available in PDF format, so there shouldn't be any concern about losing them if you cancel your subscription.

This is the kind of service that cloud computing makes possible. The Amazon Web Services blog has a brief description of how Pixily uses multiple AWS offerings to keep their prices low.

Trusting People With Your Mail

The "mixed media" nature of this company, combining real world and digital, is one of the things that makes it so interesting. There are other services like this but each are a little different. See also Earth Class Mail, which intercepts your mail before it gets to you and lets you sort it online and Scribd's Paper to iPaper service, which is free, takes its sweet time in scanning your documents and then serves ads next to them online.

Are you willing to send important paper documents to a startup company online? Privacy and security could definitely be a big concern. We are quite interested to see how Pixily works and will report back after spending some time with our new subscription.

You can watch a 5 minute screencast about Pixily here.

Netflix Comes To Xbox

Earlier this week, Netflix subscribers got a nice surprise - they no longer need to save up for that Roku box to get instant access to Netflix movies on their TV. Instead, the new set-top box for Netflix is going to be one that many people already have in their living rooms: an Xbox 360. The partnership between Xbox and Netflix will be bringing a new "Watch Instantly" feature that will appear on Xbox later this fall. In addition, a "Live Party" feature will allow people to watch movies together over Xbox Live. Well, the coolness of that feature is debatable...but still, Netflix on Xbox? Did Microsoft just win the living room from Apple?

Deep Zoom Changes the Web

Bah humbug - another browser plugin. Is that what you think? Well, like it or not the Silverlight plugin is being pushed hard. It's going to be installed on millions of HP computers and it's going to power NBC's Olympics '08 website, so it's going to become hard to avoid installing this one after a while.

If you've been paying attention to Silverlight news, you know that one of the most remarkable things about it is its Deep Zoom feature. It's definitely the coolest. It initially received attention when Hard Rock debuted their Memorabilia website. Then there was the incredible Deep Earth site (which technically didn't use Silverlight's Deep Zoom, but instead uses Silverlight plus a custom-written component created in Visual Studio). Now we have a Silverlight Deep Zoomable image of Yosemite National Park. 70 photographers, GPS-enabled cameras, 10,000 high-res photos. The results let researchers study rockfall activity and help Yosemite search-and-rescue teams with their operations by providing detailed, zoomable maps of the rockfaces. Cool? Yes, definitely.

Live Mesh

This service is rapidly approaching coolness. Mac fans have complained there's no Mesh for them, but that's only a matter of time. In the past couple of days, we've seen Live Mesh open up to all and launch a mobile web site.

Via m.mesh.com you can see your stream of Mesh news, access your Meshified folders, and move your photos, videos, and other content from your mobile device into your Mesh, instantly making them accessible from any computer, anywhere. The Live Desktop (cloud storage) offers 5 GB, but you aren't limited to meshing only 5 GB - you can mesh as much as you want. Data will sync from device to device via P2P connections, but only 5 GB are stored online for access when you're away from a device you own. You have the option to configure which files are part of that 5 GBs. Oh, and it does Remote Desktop, too.

If you haven't been able to wrap your head around Mesh, yet, this video is a killer introduction. Here, Ori Amiga demos the native Mesh feeds, WPF applications using Mesh, a Silverlight client that supports working on and offline, a custom Facebook application that syncs Facebook photos with Live Mesh, and even a Mac client that sends photos to Live Mesh. Cool? You bet.

Ori Amiga: Programming the Mesh

Your guide to this video

• 10:53: Skip to this point to start seeing the best stuff
• 19:10(ish): The developer stuff continues until 19:10ish
• 19:40: WPF demo app Family Show
• 27:01: Silverlight App PhotoZoom running offline
• 33:08: Mesh connector for Twitter
• 34:35: Mesh connector for Facebook
• 36:45: Mesh running on the Mac - photo from Photobooth synced to Mesh almost instantaneously - to both PCs and mobile!
• 43:00: Opening/editing files directly from the cloud - the cloud will be a shortcut on your desktop
• 46:09: Viewing offline RSS feeds synced to Mesh in your RSS reader

Do these innovations change your opinion of Microsoft? Are you impressed, annoyed, neutral, upset, undecided? Let us know what you think in the comments.

Author Disclosure: I also blog for Microsoft's Channel 10. I'm not a Microsoft employee, just a technology fan. This is not a paid endorsement - these are personal opinions.