Comments for Rethinking Recommendation Engines

Rethinking Recommendation Engines

2008-02-25T09:37:46Z

Over two years ago, Netflix announced a Recommendation Engine contest - anyone who invents an algorithm that does 10% better than their current recommendation system will win $1 Million dollars. Many research teams raced to attack the problem, excited by the unprecedented amount of data available. Initially quite a lot of progress was made, but then slowly the progress stalled and now teams are stuck at around the 8.5% improvement mark.

]]> In this post we argue that the improvement in recommendation engines is not an algorithmic problem, but rather a presentation issue. Respinning recommendations as filters and delivering them without setting high expectations is more likely to yield progress than crunching more data faster.

Building a recommendation engine is a complex endeavor, which we discussed here a year ago. But in addition to being a technical challenge, there are also fundamental psychological questions: do people want recommendations and if so, then when are they open to them? Perhaps an even bigger question is: what happens when the user receives one or more bad recommendations? How tolerant will they be?

Genetics of Recommendation Engines

All recommendation engines are trying to solve the following problem: given a set of ratings for a particular user, along with those of the whole user base, come up with new items that this user will like. There are many algorithms that can be applied to the problem, but all of them focus on three elements: personal, social and fundamental:

Personalized recommendation - recommend things based on the individual's past behavior
Social recommendation - recommend things based on the past behavior of similar users
Item recommendation - recommend things based on the item itself
A combination of the three approaches above

A social recommendation is also known as collaborative filtering - people who liked X also like Y. For example, people who liked Lord of The Rings are likely to enjoy Eragon and The Chronicles of Narnia. The problem with this approach is that peoples tastes do not in reality fall into simple categories. If two people share the same taste in fantasy movies, it does not mean that they will also both like dramas or mysteries. A good way to think about this problem comes from genetics. Many times we meet people who have features that we recognize and have seen in others. For example, eyes might look familiar, or lips, but it is a totally different person.

The other kind of recommendation is an item-based recommendation. The best example of this system is the Pandora music recommendation service. It works by ranking each musical piece by more than 400 different characteristic - musical genes. It then automatically matches the pieces based on these characteristics. There are challenges with tuning the algorithm to work well, but it is also challenging to apply it to other verticals. For movies, for example, you'd need to come up with ranking each movie along many scales, starting from director, cast, plot; and then obscure things like musical score, locations, light, camera work, etc. It certainly can be done, but this is complicated.

The Guy In The Garage

The complexity of the recommendation problem is due to its vast space of possibilities. Much like it's hard to figure out which exact gene is responsible for a particular human trait, it is hard to figure out which bits of the movie or music make us rate it as 5 stars. Reverse engineering human thinking is hard. Which is exactly why one of the contestants highlighted in the Wired article is relying on a very different trick to make his algorithm work.

Nicknamed Guy In The Garage, Gavin Potter from London is relying on human inertia. Apparently, the rating of the movie depends on the ratings of previous movies that we just saw. For example, if you watch three movies in a row and rate them with 4 stars, and then watch the next one which is slightly better, you will rate it 5. Conversely, if you rate three movies in a row with 1 star, then the same movie that you would otherwise rate as 5 would only get 4 stars from you.

Just when you think that this is not true, you will discover that this algorithm now sits in the 5th place and still is making progress, while other algorithms are spinning. Enhancing formulas with a bit of human psychology is a really good idea and this is where we turn next.

Replacing Recommendations with Filters

How many times has this happened to you: a friend recommended you a movie or a restaurant, so you went there all excited - but ended up disappointed? A lot! It is obvious that hype sets the bar high, increasing the chances of a miss. In math speak, this kind of miss is known as a false positive. Consider now what would happen if instead of recommending a movie, a friend tells that you are not going to like certain movie, so do not bother renting it.

What bad can come of that? Not much, because likely you are not going to watch it. But even if you do and you like it, you are not going to be experience negative feelings. This example demonstrates the difference between our reaction to a false negative and a false positive. False positives upset us, but false negatives do not. The idea of respinning recommendations as filters is about leveraging this phenomenon.

When Netflix makes recommendations, it sets itself up for a sure failure. Sooner rather than later it is going to miss and recommend you a movie that you are not going to like. What if instead of doing that, it would show you new releases and have a button: filter the ones I am not going to like. The algorithm is the same, but perception is different.

Filters in Real-Time Culture

And this idea becomes increasingly important and powerful in the age of real-time news. We are increasingly oriented towards continuously filtering new information. We do this with our RSS Readers everyday. We think of the world in terms of streams of news, where things of the past are not relevant. We do not need recommendations, because we are already over subscribed. We need noise filters. An algorithm that says: 'hey, you are definitely not going to like that' and hide it.

If the machines can do the work of aggressively throwing information out for us, then we can deal with the rest on our own. Borrowing from the spam box in emails, if all the tools around us had a button that said 'filter this for me', and maybe even had a mode where such a filter is on by default, we'd all to get more things done.

Conclusion

Building a perfect recommendation engine is a very complex task. Regardless of the method, collaborative filtering or inherent properties of things - recommendations are an unforgiving business, where false positives quickly turn users off. Perhaps applying psychology to the problem can make people appreciate what these complex algorithms are doing. If instead of recommending things, machines would filter things we definitely won't like, we might be more forgiving and understanding.

Now tell us please about your experiences with recommendation engines. Were there ones that worked really well? Would you be open to filtering instead of recommendation? Besides movies and news, where would you like to have these filters?

See also our follow-up post 10 Recommended Recommendation Engines.

Comment from Dirk Olbertz on 2008-02-25

2008-02-25T10:42:33Z

Filtering is a nice idea, because it makes more transparent what is happening. But I see recommendations as a kind of "display only a small portion of the reality".

If I'm looking for some movies by genre and there are a coupld of hundreds in it and I should us a filter on those, that process is not very usable.

But in addition to your thoughs: what about showing the recommendations, but also a link with "there are 74 movies here, that you might not like" and then show them. When we then display the reason for the negative recommendation ("your friends did not like it", or "you did not like other movies from that director", makes it perfectly transparent and helps me in choosing the right movie for me - something that I obviously will not find out, until I watched the movie, but I know that when doing the choice :-)

Comment from Data Mining and Knowledge Discovery Search Engine on 2008-02-25

2008-02-25T11:10:13Z

Data Mining and Knowledge Discovery Search Engine

This is a search engine all about Data Mining and Knowledge Discovery, including related International Conferences, Journals, Companies, Blogs, People, Research Institutes, softwares, products, courses, tutorials, papers, codes and etc..

http://www.google.com/coop/cse?cx=006422944775554126616%3Aixcd3tdxkke

Comment from TS on 2008-02-25

2008-02-25T11:13:41Z

Filtering sounds like a good idea at first glance but it made me think. I feel it is potentially dangerous in term of filtering out alternative content.

Imagine you are a really strange subject who likes things a very little percent of other users like. A filtering engine would say ..."hah, since almost nobody likes this, most probably you won't like it too"... and you would never even see that content listed (even if it might be of interest to you).

Just a thought :)

Comment from Amit on 2008-02-25

2008-02-25T11:58:29Z

The theory that "False positives upset us, but false negatives do not"
maybe true for films, but it is not true in the general case. Imagine I'm investing in Apple stocks, than I'm willing to tolerate some news that are irrelevant about the company. But I wouldn't want to miss any story that may imply a dip in the stock price.
The psychological value of false positives and false negatives is determined by the relative price (time/money/effort) one have to pay for each.

Comment from IdeaTagger on 2008-02-25

2008-02-25T12:40:19Z

The issue with automatic filtering is one of trust. What I have found when I use fee filters like FeedHub etc is that I keep going back to my original feeds to make sure the filters haven't missed out anything interesting - talk about addiction. Sad I know, but probably true for a lot of people. Oh and each time I do find something that the filters missed, it degrades my trust in the filters and makes me more likely to check again.

BTW, I think social recommendation of the "people who liked this also liked" variety is good. On the other hand, I think the "your friends liked this so you might too" kind overestimates the significance of friends. However, "your friend Joe thinks you might like this" is probably quite interesting, especially if I can set whose recommendations I am interested in and what number of such recommendations should trigger a notification to me. So on Facebook or something you would ask users "which of your friends would like this book/movie?" and use that as a basis.

Comment from preetam mukherjee on 2008-02-25

2008-02-25T13:37:45Z

Alex,

Thank you for highlighting an important aspect of the evolving Internet.

Recommendation engines typically work off a given social (+/-) contextual construct.

However, the dynamics of Internet hasn't scaled in the last few years. They've evolved. There have been disruptions. So while Amazon, Netflix, et al were commendable in achieving their objectives in a yester-year, we now DO need to think about recommendation engine in a very different light.

Three things to think about are content(nature, niche, information), audience(individuals/groups/masses, preferences, tastes, habits), and context(who, when, where..and hopefully why).

We're a startup out of Berkeley working on a recommendation engine for video that we've been testing out extensively. It's based on the above model, and we're focusing our efforts on personalization across the video domain.

"Bad recommendations" always exist. You can't have a perfect recommendation engine. What one can do is introduce learning mechanisms to understand why a recommendation was bad, and what a user chose in lieu of. You won't develop serious wisdom from just one observation on alternate choice, but over time/aggregated use, you'd be surprised how few those "bad recommendation" numbers become!!

Also, part of the problem with rec. engines is that they're mostly active-interaction based(ratings, purchases, et al). But there's a lot you could learn from passive transactions(user flow, navigation, alternate selections, de-selections...) which can be used to provide a complementary inferencing basis for your recommendations.

Still in stealth mode, so can't go into more detail, but hope all that helps. Let me know if I can clarify something further.

-p

Comment from Scott on 2008-02-25

2008-02-25T15:35:05Z

Great post. This reminded me that in computer science and math proofs, one of the most successful techniques is to solve the *inverse* of the problem you're working on -- which is sometimes significantly easier -- and then exploit that result. It's a counterintuitive approach, but often very effective.

It's also a technique that could be used more in business brainstorming, particularly for start-ups and disruptive innovations.

Comment from p-air on 2008-02-25

2008-02-25T17:14:29Z

Thanks for bringing these methods into a common context. While I'm far fm an authority, you may have mischaractized social recommendations (or collaborative filtering) in your description if it. Specically:

"For example, people who liked Lord of The Rings are likely to enjoy Eragon and The Chronicles of Narnia. The problem with this approach is that peoples tastes do not in reality fall into simple categories."

The collaborative filtering services that I'm familiar with are looking at this statistically. In other words, people who liked Lord of The Rings are likely to enjoy Eragon and The Chronicles of Narnia, because others who liked Lord of The Rings also bought or rented or watched the other two movies. It has nothing to do w/tastes. By the same token, one could make such recommendations across categories because this is a clustering exercise that says that people who bought A, B & C also bought D, therefore if you bought A, B & C then you might also be interested in D. When these systems don't have enough data on a specific item, they might move up a level of abstraction to a topic or interest level but that's only when the item level data isn't there. It doesn't take long to have enough item level data to increase the precision of recommendations.

Not that this was the point of your write-up, but it seemed like a point worth clarifying.

Comment from preetam mukherjee on 2008-02-25

2008-02-25T19:46:51Z

good point, p-air.

however, "social recommendations" can be(and is) used loosely.

It could refer to: "your friends liked...".
Or it could refer to the collaborative filtering scenario which you've outlined accurately.

note that collaborative systems examine tastes based on item-level data(implicit + explicit metadata), as well as using aggregate information(that's where the 'collaborative' part comes in)

Comment from Arnie on 2008-02-25

2008-02-25T20:11:23Z

We actually have a patented recommendation system that no one really knows about. We tried to get it out before the bubble burst but haven't done anything with it in a few years.

It is a ground floor patent that is basically the opposite of collaborative filtering. It correlates the items rather than the people and is incredibly accurate. In fact, more accurate than the Netflix system - BTW... no one will win their contest either, unless they get very, very lucky.

If anyone reading this post is interested in acquiring or licensing our patent, contact arnie [at] cox.net.

Comment from Matt Gillooly on 2008-02-25

2008-02-25T22:03:13Z

To extend on Amit's point, I think a preference for false positives versus false negatives comes down to the abundance of acceptable results, and the cruciality of any individual item.

In the case of movies, there are many more good-enough picks than most of us will ever watch, and few of them are particularly crucial to watch. There is little cost associated with missing one particular sure-thing, but a noticeable cost associated with letting any poor choices through. I think most consumer entertainment/food/product recommendation problems fall into this category.

In the case of financial or medical info, however, there are individual items that are crucial enough to merit wading through a larger set which may contain false positives.

Even within a given vertical, this idea has applications to social recommendation systems. Results from my inner circle of trusted friends could be considered more crucial than people I trust less.

Comment from Fabio De Bernardi on 2008-02-25

2008-02-26T00:12:22Z

Getting it right with recommendations is a tough job. Especially if your focus is multiple and not only movies, or book, music, etc.
While applying these theories in my startup we thought about using reverse-recommendations (or false negatives) to enhance the information set delivered.
It's tricky though to get it right and to let people understand that there's real information in things you shouldn't - in theory - like.
Especially in social networks the aggregation of users around things they didn't like could be pretty powerful to exploit their similar traits. If then you add their interests on top of it you could come up with some pretty nice association. Of course it can't be the only aspect to consider but it could enhance the understanding of people/products/music/etc connections.

Comment from Daniel Barnett on 2008-02-25

2008-02-26T07:03:49Z

Nice idea to look at the opposite. I thought that the competition was scored on accurately predicting user ratings instead of only providing recommendations.

I think it would be better not to favour either recomending or filtering but simply to give the user their 'expected rating' for each movie as well as a indication of the confidence of the expected rating. Then they could do what they wanted with that information. For example

Movie : Jaws
Average Rating : 4.2
Your Expected Rating : 2.9 - 3.7 (80%)

Something like that.

Comment from Joe McCarthy on 2008-02-27

2008-02-27T20:59:30Z

I've tried twice to leave a trackback here from a blog post I wrote in response to this one, but did not succeed.

A brief summary of my post, Re-rethinking Recommendation Engines: Psychology and the Influence of False Negatives, is that I like the article, especially its characterization of different kinds of recommendations and its emphasis on the importance of psychological factors in technology design, but have concerns about its promotion of the acceptability of false negatives, which could reduce serendipitous discoveries and lead to more harmful consequences from overly aggressive filtering.

Comment from Yuri Syuganov on 2008-02-27

2008-02-28T02:51:09Z

It looks for me netflix's (and few other sites, like flixster) approach is: movies are entertainment. Yes, it is for the majority of people. And they are OK with filtering out "not worth your 2 hours/10 backs" list for what currently is in movie theatres.

There are also less big but more demanding audience of people around for whom movies are a form of art. (And the mix of those, of course.)

I think recommendations those two audiences would appreciate should be based on different set of criterias. The "art lovers" don't care much about "blockbusters" lovers recommendations no matter if those are their friends or not.

I agree with the idea that presentation is the huge part of recommendations perception.

...My expectation is: the next generation recommendation system will first run an adaptive test creating user's profile and then showing not only recommended movie titles but give a clue why those were suggested (not as another link but inside the list).

Another thought: it's important to distinguish between "I think you'll love it" and "based on your choice it worth seeing that movie from the same director which is even better than this his latest one".

So adaptive test should show movie titles and produce next questions or titles to rank -- to find out "why you said you might recommend this movie to someone else".
Example: my wife loves "House, M.D.". She says -- mostly because of Hugh Laurie character's personality. But at the same time she hates "A Bit of Fry and Laurie" -- for (her word) "stupidity" of those sketches. So it's not about the person H.Laurie, but about dialogs (script writer) and his acting while he's in his mid 40s. So based on someone's love to "House, M.D." we might want to find out if they'd like recommendations about other popular TV series, or TV series about doctors, or other movies Laurie played in, or other shows with smart-ass politically incorrect dialogs, or detective-type storytelling. And based on users' profiles those users recommendations (not in 5-stars scale but in 1-bit: would you or would you not recommend it because of this particular feature) will have more or less weight in computing recommendations to other users who share the same likes/dislikes -- not based on titles but on which particular things they recommended these titles for.

Profiling system should find out about each user how much (s)he sees and loves "art" in movies (everything that distinguish movie from literature, theatre and plain old TV); what genres he has preferences in (some don't like horror movies at all, but wouldn't mind to see one or two considered the landmarks in the genre, others hate high-school girlie stories, but will be OK to see their favorite actrees' early work); which persons (directors, actors, script writers, cameramen/cinematographers) might interest him (I'm curious about all Todd Solondz movies or all where Kristin Scott Thomas played); how omnivorous he is (I enjoyed "Shoot 'Em Up" and "Nostalghia", "When Harry met Sally" and "Death Proof" -- wouldn't be surprised if someone said those are way too different tastes -- but I liked all of them); how much he knows/saw already; etc. And questions to ask/lists of titles to present for ranking should be created not automatically but by alive persons/movie critiques to reflect the best achievements in movie-making history. And after we found that one particular person saw everything worth seeing in this genre/from this director/with this actor -- we'll need to switch from recommendations based on the fact too many people recommended this title to some rare stuff which still might interest him.

So based on a title I recommended I'd like to see at least:
- "more titles from this director" (in back chronological order with rating from other users computed to reflect my preferences);
- "more titles with this lead actor(s)" (his early years; his mature years; his best/worst movies...; his director's work);
- "more titles with the similar story twist" ("she's older than him, he's a kid"; "monsters in New York, shot with handycam", "good cop/bad cop -- everyone dies at the end" -- OK, skip the last one, no spoilers);
- "other pictures won the same fistival awards that year", etc.

So: don't try to guess what I will love. Suggest what might worth my time and list the factors why you think so.

Comment from Andrew Lee on 2008-02-28

2008-02-28T23:45:12Z

All of this in relation to your previous posts on the semantic web, seem to show that perhaps, the future of "intelligent agents" is not what we're interested in immediately, but rather a future where we receive a stream of information that we can click a button and filter/highlight the things that would interest us.

Odd, I wonder if there would be ways instead of SEO, there would be Recommendation algorithm optimization (RAO) for all different types of products.

Comment from metropol.myopenid.com on 2008-03-05

2008-03-05T08:08:08Z

Hi.

About the presentation (interaction) vs evaluation (playlist) aspect of it.

One. It seems that, rather than the inherent or objective, computational relevance of recommended items and or ways of presenting them, what is more important is who is the vehicle, the recommender and how I value her, in absolute or relative terms.
For example, I will value a recommendation by a person whose (movie) taste I appreciate, much more than any score a recommendation engine could provide.

Two. Recommendation engines can do a useful job in drastically reducing the hypothesis space, but still they don't deal with the possibly more critical aspect of how to have the user to buy (no pun intended) into some of the recommendable items. That is the interactive, argumentative aspect of a recommendation.
Maybe more resources should be devoted at seeing the problems in terms of what makes a good (way of) recommendation, rather than what makes recommendation possible (which seems to be the focus of the common elements cited in the article).