Should YouTube Scrap its Ratings System and Rely on Implicit User Data?
Last week YouTube blogged that it is considering moving away from the familiar 5-star system of reviews. According to YouTube product manager Shiva Rajaraman, the stars system is being used bluntly by the majority of YouTube users - most give videos a perfect 5 star rating. Rajaraman noted that "when it comes to ratings it's pretty much all or nothing."
When you also consider that the wisdom of the crowds is often dominated by small, powerful groups, then the validity of user ratings is further called into question. So why not just get rid of explicit user ratings and use implicit recommendations instead?
YouTube graph showing the dominance of full 5-star ratings
YouTube wants to know if a thumbs up/thumbs down system would be be more effective (two options), or even just favoriting (one explicit action to say you like an item).
However possibly a better option is to remove explicit ratings altogether. Does YouTube even need to ask its users for ratings, given the wealth of user interaction data it has?
Earlier this year, ReadWriteWeb profiled some sophisticated recommendation technologies which rely on implicit user data. Many of these systems track user data and, with a set of (usually proprietary) algorithms, come up with recommendations for users. This type of system could well replace ratings altogether in YouTube. While YouTube probably already makes use of the ratings data in its recommendations, as noted above such data is typically unreliable and not very valuable.
As an example of how this could work on YouTube, here is our description of Baynote's recommendation system:
"Baynote observes real-time user behavior on a site and looks for implicit, emergent patterns. It uses collective intelligence and an affinity engine to analyze the data. Common behaviors which it tracks include page refers, queries, mouse movement, time spent on a page, peer behavior (see note about communities below)."
Other similar recommendation technologies we've profiled include MyBuys, ATG and richrelevance.
Are explicit user ratings still valid in consumer apps such as YouTube and Amazon? While we're arguing that implicit recommendations data could enable YouTube to scrap user ratings altogether, on the other hand products like RateItAll are still built around the star system. Let us know your thoughts in the comments.
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
I like nuanced ratings for things like videos, but acknowledge most people just do like or dislike. Unfortunately, unless it was god-awful I never tend to just dislike something as a rating. So I mark up good things, and rarely mark down when using thumbs up systems. On the other hand, I rarely give videos 5 stars in nuanced ratings. I do no rating if I hated it, or 3 to 5 stars. Oddly, I can't recall ever using the 1 or 2 stars. Possibly, cause if the videos that bad, I didn't finish it.
At ESPN, we went with Like/Don't like. That's all fans are looking to do. They either love or hate something.
Youtube could normalize the data per-user. If someone is just 1 or 5 without anything inbetween the software could internally normalize that to a range of 2-3, whereas people who use the full range would be allowed 1-5. That would be a quick hack that uses the existing data.
That said, Reddit and Digg use simple yay/nay voting and it seems to work well.
They should show the distribution on each video (i.e. 1 stars: 5, 2 stars: 10, 3 stars: 4, 4 stars: 10, 5 stars: 30) except do it with a LIKE, NEUTRAL, DISLIKE (3 star system) since most videos will get either a good rating or a bad rating or no rating at all.
Explicit user ratings don't work because too many users give a perfect 5 star rating and it's very easy to game the system.
Implicit recommendations based on sentiment analysis are more useful. We're working on this at http://rankspeed.com
Richard,
you are totally right that there is undoubtly a strong correlation between explicit and implicit feedback collected trough user interaction. Theoretically the explicit feedback can be removed or altered to a similar pendant. Youtube's graph indicates that a Facebook-style thumbs up feature would be a simplified version that drops almost no knowledge. Explicit ratings are a nice feature to double-check the implicit rating tracked.
However, I think that regardless of explicit ratings, the collection and analysis of implicit feedback is a must because it contains objective insights which is valuable for recommendation generation. This year I co-published a semantic user modeling approach (http://www.springerlink.com/content/t617573358j43077) at the UMAP (no 1 conference on user modeling and adaptive personalization http://umap09.fbk.eu) which goes a step further in that realm because we semantically augment feedback data on the client side before sending it to the server. Doing so, valuable semantic knowledge is preserved in contrast to conventional user tracking mechanisms.
Cheers,
@alexkorth
I’ve always been a bit against the rating system, but it’s also set the standard for a lot of other sites out there which now have the exact same kind of thing. The only site that comes to mind in which the five-star system actually works somewhat well is Amazon. Other than that it’s pretty worthless.
I would go for the two-vote option. I’m not going to favorite videos that don’t pertain to me just because I think they’re funny. Once I absorb someone’s content, unless it’s viral or it is mentioning me in it, I usually don’t need to return. So why add that video to my favorites if it really isn’t?
After hours of watching video on YouTube, one quickly realizes that the rating system leads to a lot of wasted time. Many highly rated videos are just not good.
That realization lead us to put a two-vote skin on YouTube and other video sites. It's not perfect yet, but you can focus on the best videos and waste time more efficiently :)
Great post, Richard!
We’re glad to see YouTube is examining its rating system with an eye on delivering value to its community and look forward to seeing how it evolves from here. Ratings and user generated reviews, though often misleading, have become an expected part of the online experience and encourage deeper engagement so we don’t view explicit vs. implicit user feedback as an either/or scenario. As YouTube retools its system, hopefully they’ll look at better ways to glean insight from the silent majority of their visitors versus only the loud minority of folks who actively post feedback and rank videos. You and the ReadWriteWeb readers might be interested in checking out my paper on this very issue, titled “7 Deadly Biases”: http://www.baynote.com/resources/white-papers/deadly-biases/deadly-biases-2008-10-30.pdf
Scott
UserVoice has a good system - you get a finite set of points (say, 10) and can assign them as you like. The more you like something, the more points you give it. You get your points back in various ways. In YouTube's case perhaps your points could return gradually, for instance on a monthly basis. It's simple, transparent, doesn't involve any complicated programming, and it's fair.
I think it's always nice to have a rating system, but I would vote for simplifying things. Looking at the graph speaks for itself and supports the "all or nothing" mentality.
I like the Digg and Facebook model though where people can either say "it's cool" or "it sucks". It's always nice to know what people are digging that way.
To me the rated system is a waste of time.
So, rated systems are always going to be skewed, but the thing about those types of ratings is that if there is a trend, all the data is still valid. So what if someone always rates a 5, as long as they always do it to the same sorts of things. I learned in communication studies to ignore the answers and look for the trends. If there wasn't a trend, the information meant squat.
I am a fan of rating things and do it all the time. I'm an active member in the www.sazze.com community and sometimes wish I had a "dislike" button for things in facebook. But as is, I think keeping the ranked position would be ideal for youtube but I could see it being okay adding a thumbs down/up system as well.
Good post!
Guys, I am a PhD student doing research in recommender systems. In first year I did work in rating based data sets like MovieLens, Netflix, etc. The main idea was to decrease error in predicting, would a user like this item and how much he would like (e.g. {1,2,3,4,5} in 5-level scale)?
In past, these technique may work, but after Netflix prize competition, researchers have invented much techniques for predictions. I am finding little gap in "rating based systems".
I have to choose a good topic to research. What you guys think, what would be a good Future research topic/challenge in recommender systems?
Any Help is appreciated!