We had the opportunity to speak with Satnam Alag, author of the recently published Collective Intelligence in Action, about what makes for a good recommendation system, where the technology is heading, and why Netflix is finding it so hard to improve its own system.

ReadWriteWeb: In our recent post about Netflix, we identified four main approaches to recommendations: Personalized recommendation: based on prior behavior of the user; Social recommendation: based on prior behavior of similar users; Item recommendation: based on the item itself; And a combination of all three. Do you agree with the four approaches we laid out in our article?

Satnam: Those four categories are pretty comprehensive. I present an alternate classification of recommendation systems in my book. I lay out two fundamental approaches. The first approach, item-based analysis, determines items that are related to a particular item. When a user likes a particular item, related ones are recommended. The second approach, user-based analysis, first determines users who are similar to that user.

Further, there are two main approaches to finding similar items and similar users. For the first, content-based analysis, content associated with the item, especially text, is used to compute similarity. In the second, the collaborative approach, actions such as ratings, bookmarking, and so forth are used to find similar items. For the second, user-based analysis, a number of approaches have been taken, including ones based on profile information, user actions, and lists of the user's friends or contacts. Of course, you can combine any these item/user and content/collaborative approaches to build a recommendation system.

The dimensions of the particular item and user space are helpful in deciding whether to use an item-based or user-based approach. Typically, an item-based approach is used to bootstrap one's application when the number of users is small. As the user base grows, the item-based approach is augmented by a user-based approach.

ReadWriteWeb: Other than Amazon and Netflix, which Internet companies have most impressed you in their implementation of recommendation systems?

Satnam: Other than Amazon and Nextflix, Google News' personalization is my personal favorite. Google News is a good example of building a scalable recommendation system for a large number of users (several million unique visitors per month) and a large number of items (several million new stories every two months), with constant item churn. This is different from Amazon's, whose rate of item churn is much lower. Google decided to use collaborative filtering for its recommendation system mainly because of its access to the data of its large user base and because this same approach could be applied to other applications, countries, and languages. A content-based recommendation system perhaps could have worked just as well, but may have required language- or location-specific tweaking. Google also wanted to leverage the same collaborative filtering technology to be able to recommend images, videos, and music, for which it's more difficult to analyze the underlying content.

Among start-ups, my personal favorite is the one we are developing at my current company, NextBio. It's not available yet but should be next month. The key point about this particular recommendation engine is its strong use of an ontology, similar in concept to tags, to develop a common vocabulary for items and users. The system then makes use of profile information and user interactions, both short- and long-term, to provide recommendations. The system leverages both item- and user-based approaches.

ReadWriteWeb: What commercial opportunities do you forsee with recommendation systems over the next few years?

Satnam: A good personalized recommendation system can mean the difference between a successful and a failed website. Given that most applications now invite users to interact and to leverage user-generated content, new content is being generated at a phenomenal rate. Showing the right content to the right user at the right time is key to creating a sticky application. I would be surprised if most successful websites did not leverage recommendation systems to provide personalized experiences to their users.

ReadWriteWeb: Your book includes a discussion of collaborative filtering. Can you tell us a bit about how this fits into the overall picture of recommendation systems?

Satnam: In recent years, an increasing amount of user interaction has provided applications with a large amount of information that can be converted into intelligence. This interaction may be in the form of ratings, blog entries, item tagging, user connections, or shared items of interest. This has led to the problem of information overload. What we need is a system that can recommend items based on the user's interests and interactions. This is where personalization and recommendation engines come in.

In my book, I take a holistic view of adding intelligence to one's application, a recommendation engine being one way to do it. The book focuses on both content-based and collaborative approaches to building recommendation systems. It focuses on capturing relevant information about the user, information from both within and outside one's application, and converting it into recommendations. One of the things you mentioned in your write-up on recommendation systems is that you would like to apply such a system to your website to recommend things to users. Someone reading my book should be able to create such a system using the techniques I demonstrate.

Next Page: Satnam's thoughts on the Netflix Prize and whether the 10% mark will ever be reached.

ReadWriteWeb: Netflix is offering $1 million to the team that can improve its recommendation algorithm by 10%. It's been over 2 years now, with the leading company at 9.63%. There is some skepticism, though, that 10% will be reached anytime soon, because now the contestants are making only incremental progress. Do you expect the 10% mark to be reached soon?

Satnam: Netflix's recommendation engine, Cinematch, uses an item-to-item algorithm (similar to Amazon's) with a number of heuristics. Given that Netflix' recommendation system has been very successful in the real world, it is pretty impressive that teams have been able to improve on it by as much as 9.63%. Of course, the Netflix competition doesn't take into account speed of implementation or the scalability of the approach. It simply focuses on the quality of recommendations in terms of closing the gap between user rating and predicted rating. So, it isn't clear whether Netflix will be able to leverage all of the innovation coming out of this competition. Also, the Netflix data doesn't contain much information to allow for a content-based approach; it's for this reason that teams are focusing on collaborative-based techniques.

The challenges to reaching the 10% mark are:

Skewed data: The data set for the competition consists of more than 100 million anonymous movie ratings, using a scale of one to five stars, made by 480,000 users for 17,770 movies. Note that the user-item data set for this problem is sparsely populated, with nearly 99% of user-item entries being zero. The distribution of movies per user is skewed. The median number of ratings per user is 93. About 10% of users rated 16 or fewer movies, while 25% of users rated 36 or fewer. Two users rated as many as 17,000 movies. Similarly, the ratings per movie are also skewed: almost half the user base rated one popular movie (Miss Congeniality); about 25% of movies had 190 or fewer ratings; and a handful of movies were rated fewer than 10 times.

The approach: The winning team, BellKor, spent more than 2,000 combined hours poring over data to find the winning solution. The winning solution was a linear combination of 107 sets of predictions. Many of the algorithms involved either the nearest-neighbor method (k-NN) or latent factor models, such as SVD/factorization and Restricted Boltzmann Machines (RBMs).

The winning solution uses k-NN to predict the rating for a user, using both the Pearson-r correlation and cosine methods to compute the similarities, with corrections to remove item-specific and user-specific biases. Latent semantic models are also widely used in the winning solution.

The BellKor team found it important to use a variety of models that compensated for each other's shortcomings. No one model alone could have gotten the BellKor team to the top of the competition. The combined set of models achieved an improvement of 8.43% over Cinematch, while the best model -- a hybrid of k-NN applied to output from RBMs -- improved the result by 6.43%. The biggest improvement by LSI methods was 5.1%, with the best pure k-NN model scoring below that. (K for the k-NN methods was in the range of 20 to 50.) The BellKor team also applied a number of heuristics to further improve the results.

The BellKor team demonstrates a number of guidelines for building a winning solution to this kind of competition:

• Combining complementary models helps improve the overall solution. Note that a linear combination of three models, one each for k-NN, LSI, and RBM, would have yielded fairly good results, an improvement of 7.58%.
• A principled approach is needed to optimize the solution.
• The key to winning is building models that can accurately predict when there is sufficient data, without over-applying in the absence of adequate data.

The final solution will be along the same lines, combining multiple models with heuristics. Contestants will probably reach the magic 10% mark in the next year or two.

ReadWriteWeb: Some people think the 10% mark can't be reached with algorithms alone, but that the "human" element is required. For example, ClerkDogs is a service that hires actual former video-store clerks to "create a database that is much richer and deeper than the collaborative filtering engines." It's a similar approach to that of Pandora, which has 50 employees who listen to and tag songs. How far do you think algorithms can go in making recommendations?

Satnam: Recommendation systems are not perfect. A number of elements go into making successful ones, including approach, the speed of computing results, heuristics, the exploration and exploitation of coefficients, and so on. But it has been shown in the real world that the more personalized you can make recommendations, the higher the click-through rate, the stickier the application, and the lower the bounce rate.

Using humans to form a rich database for recommendations may work for small applications, but it would probably be too expensive to scale. I don't see them competing against each other, human versus machine. Even with human/expert recommendations, one first needs to find a human/expert with tastes similar to those of the user, especially if you want to go after the long tail.

1. The Wisdom of Crowds and Law of Large Numbers suggest that as communities grow, not only does a large (diverse, independent, etc.) community make better decisions than a handful of editors, but the larger a community gets, the better its decisions will be. Therefore, we can hypothetically create a Collaboratively Filtered newspaper, television channel, radio station, etc., which would be better (for the community) rather than any other arbitrarily selected medium. In fact, as we will see, services like Digg, YouTube, and Last.fm, are trying to do exactly that - (CF) based media outlets.

2. The second principle of Collaborative Filtering suggests that in any such large community, with enough data on individual participants and on how the individual participants collaborate or correlate with each other, we can make predictions about what these users will like in the future based on what their tastes have been in the past, i.e. develop a collaboratively filtered recommendation engine. This, of course, relies on the fact that people's interests, preferences, and ideologies don't change too drastically over time.

The two aspects of the (CF) system result in two very different and important results.

The first gives you new, interesting, entertaining, and newsworthy information as judged by the community (in a way this is content that is the average of the interests of the entire community) and a good example of this is Digg's front page. Not all the content will be directly relevant to your tastes and in fact some of it will be completely irrelevant to you. However, as the community grows and becomes more diverse and independent, the average news story promoted to the front page will be of interest to the average community member. Not satisfied with averages? This is where the second aspect comes into place.

The second aspect of the (CF) system collects information on what kind of content and commentary you like and dislike, and based on your submission and voting habits, it does user-data-profiling. This user profile helps the site recommend content that has been submitted by users (or from sources) you generally agree with and find interesting, as well as topics that you usually vote up and tend to comment on. What this means is that by collecting enough information on how you interact with the site and with other users, the (CF) system can recommend content to you. The system finds the content and deliver it to you rather than it requiring you to scout for it. Furthermore, the more you use the recommendation system and vote up or down, the better it becomes with its recommendations.

The important thing, one that not many social sites realize, is that a (CF) system that doesn't automatically match content to your preferences, is inherently flawed. The reason for this is simple: Unless you can achieve perfect diversity and independence of opinion, one point of view will always dominate another on a particular platform. The dominant point of view on the social web is a left-leaning one, and without the ability to get the most appropriate pieces of content to the people that care most about them, the right-wing point of view gets buried almost every time.

A perfect example of this was the Ron Paul supporters and the ease with which they were able to manipulate the social news sites. Now if you could match the right-wing viewpoint to the right-wingers, and the left-wing viewpoint to the left-wingers, and get both points of views across to people that are interested in healthy debate rather than partisan politics, you're getting closer to the ideal system. A filtering system with preference-based recommendations, in essence, is the future of the social web.

Who is using what system?

The (CF) system is without a doubt the lifeblood of the social web. Even though different platforms apply it to varying extents, the system is still there at the core, and the social web would look more like rush hour in downtown Lahore if the community wasn't actively policing the traffic.

Social News

In the social news space, Digg and Propeller just use the system insofar as the front page is concerned (although Digg is set to release their recommendation engine this week). Once the content is promoted to the front page, the system's job is done. The system works in that you get rid of spam and unoriginal thought, but it isn't the best because it relies on averages rather than direct preferences of each participant. While these sites try to catch up and develop recommendation engines of their own, Reddit and StumbleUpon have leapfrogged them for a while now by having recommendation engines in place. These two sites also have similar concepts of a community front page (based on the average interests of the average community member) but they enhance your experience and incentivize increased participation by using your history of likes and dislikes to deliver the most high-quality and most relevant content to you. Furthermore, the normalization of Reddit's front page shown how a one-front-page-for-all approach forces conformity and dilutes the individual experience, whereas normalization ensures that each user controls how content is distributed to him or her.

Ultimately, even though there are some sites with little or no filtering (Slashdot, Fark, etc.), sites that use their (CF) based recommendation engines will continue to diminish the importance of active filtering from upcoming submission queues and improve the quality of user experience on an individual level.

Video Streaming and Sharing

Online video sites hosting and sharing sites are not much different. Site's like YouTube have multiple filtering mechanisms that often perform the same functions without requiring votes per se. Viewability, for example, is determined by:

1. Number of people currently watching a video

2. Number of comments on a video

3. User ratings and favorites.

The problem with impressions-based system (like the one used by now understandably dead content aggregator Spotplex) is that just because you viewed something or commented on something doesn't mean that it's good. In fact, there are dozens of YouTube videos that I click on, don't like them and then close the window (I see other people writing negative comments in poor English but I doubt that helps either). Some other sites like Break and Funny or Die use a StumbleUpon-like up/down voting system to determine what gets promoted to the front page. Again, while there are options to view similar/related videos and more videos from a user you like, there is no recommendation system using your rating and favoriting habits (and tags you like).

Blogging and Microblogging

For the most part, blogs use a combination of most viewed, most linked, most commented, and highest rated, as mechanisms for displaying content that you might like. While this is a better idea than letting people go through trial and error, it doesn't ensure that every visitor will be happy with what they see. For example, two very different posts on two entirely different topics can be the most viewed posts on your blog, and I might like one and not like another. At the same time, one has to wonder, at what point is it economical or time-efficient to start monitoring each individual user?

StumbleUpon solves this problem for the 'big guys' by letting you StumbleThru one site for the content that you might like the most. The feature, however, is not available for all sites yet.

Most Microblogging sites, unfortunately, have no filtering system at all. The signal to noise ratio debate rages on with respect to Twitter and its ilk. FriendFeed, however, launched a rudimentary recommendation feature that simple displays the top 'liked' and commented links.

Photo Hosting and Sharing

When I was thinking about the concept of (CF) systems, photo-sites like Flickr and Photobucket weren't even on my radar. Part of the reason for this is how most people I know use these sites, i.e., primarily for hosting and sometimes for finding creative commons images for embedding on their sites. I was, however, quite pleasantly surprised to see that Flickr has gone a long way to help people explore and discover excellent photography.

The feature that most people are probably familiar with is Interestingness. The feature is quite robust. It takes into account things like where the referral traffic to the image is coming from, who is commenting on it and when, who marks it as a favorite and how many people like it, among other more nuanced things. But in addition to that, the site also has other great features such as exploring based on geotagging on a map of the world, popular tags, subject-based and quality-based groups, camera finder, and most recent uploads.

The only thing left to add is a 'photos you might like' based on photos you have liked and commented on.

Music Streaming and Discovery

The best implementations of a Collaborative Filtering (CF) system along with a preference based recommendation/discovery system that I have seen are always on music streaming and discovery sites. The implementation on Last.fm for example, is almost perfect in my opinion. First of all, whether you use their online streaming widget or use their desktop software, they monitor every single song you listen to and aggregate that data. They also track how artists jump on and fall off your radar on a week to week basis. They use that data to make specific recommendations and automatically create a radio station for you that plays Last.fm's recommendations for you based on what you like.

While that in itself is more than enough, they don't stop there. They have another radio station for you that plays songs you usually like to listen, they show you what the entire Last.fm community is generally listening to, what your friends are listening to, and what your friends are recommending. It is a very robust system for aggregation, filtering, and recommendation. Here's how the recommendation engine works:

As you can see, they look at the musicians I listen to a lot and then recommend people that are either similar in sound or people who were influencers of or influenced by my favorites. These are followed by recommendations from friends and music-based groups on the site.

So, collaboratively filter and recommend or die?

These are only some of the major players that have embraced (CF) and personalized recommendations - Netflix and Amazon come to mind among others. As you can see from above, it is certainly possible to have a good collaborative filtering system without a recommendation engine (as seen in Flickr). It is optimal, however, for the users (because their experience is better) and your site (because users will participate more often and generally be happy with your product) if you throw in some recommendation system a-la Last.fm, the most robust of the lot by far.

This is a guest post by Muhammad Saleem, a social media consultant and a top-ranked community member on multiple social news sites. You can follow Muhammad on Twitter.

How Well Does it Work?

Here's how Aggregate Knowledge describes the system for BusinessWeek.com:

"When a reader clicks on a breaking news story on the site, the Aggregate Knowledge Pique Discovery Window automatically provides user-driven content suggestions in the form of “More from BusinessWeek.” These suggestions are based on what visitors are actually reading across BusinessWeek.com."

I clicked some stories on the BusinessWeek.com homepage, and noticed a "More from BusinessWeek" list of links to the right of each story. However, none of these links seemed very relevant to the story. Check out this example from a story about Apple iTunes:

No Apple or even tech stories are linked to. Here's another example - about Russian police visiting BP offices. Curiously, this one lists an Apple story!

No Actual Content Analysis?

So based on my tests, it doesn't seem like there is much - if any - semantic analysis of the page content in order to come up with the "More from BusinessWeek" links. Reading between the lines of the AK quote above, this discovery system is based on clicks and not content.

It appears as if this is collaborative filtering - i.e. users who clicked X also clicked Y. This is basically the system that Amazon and Netflix use. For Aggregate Knowledge, collaborative filtering is still going to give interesting results. But how is it better than - for example - the 'Related Entries' plugin that we use here on ReadWriteWeb (which is based on tags, and so is much more closely aligned to the content itself). See bottom of this post for an example.

Surely for media sites a content discovery system that analyzes the content of a page, such as Reuters Open Calais does, would give better results. Please let us know your opinion in the comments.