Google Hires People for Feedback on Search Results - Is Anyone Surprised?
TechnologyReview has published an interview with Google's Director of Research, Peter Norvig, where it's acknowledged that the search giant hires individuals to look at search results pages and provide their judgment about the quality of the results.
This one part of the interview lead to a thoroughly researched blog post by Saul Hansell at the New York Times, one at watch-dog blog Google Blogoscoped and will probably be written about in many more places before the day is through (heck, including here). I can't stop wondering, though - is this really news or at all a surprise? It's both fascinating and mundane at the same time.
Here's what Norvig actually said:
Another way we do it is to randomly select specific queries and hire people to say how good our results are. These are just contractors that we hire who give their judgment. We train them on how to identify spam and other bad sites, and then we record their judgments and track against that. It's more of a gold standard because it's someone giving a real opinion, but of course, since there's a human in the loop, we can't afford to do as much of it.
This is a matter of great importance, as Google holds so much power over our collective thinking and knowledge these days.
Does this mean that these peoples' judgment is weighed heavily in determining the results? Are their thoughts used to improve the algorithm or are they acted upon directly? Does the introduction of a human element smack of some unfairness that an algorithm, though created by humans as well, lacks?
It's not very clear at all just what Norvig means by his statement. For all its prestige and access, the publication that did the interview appears to have done so by emailing a predetermined list of questions to the subject - at least there was no follow up on this very interesting response. As such, there's any number of ways readers could take it. As one commenter on the NYT says, "There’s an important difference between judging your algorithm’s results versus incorporating ratings into your algorithm." The next commenter, though, says that the first is missing the point. "everyone knew that Google was continuously assessing their algorithm, the fact that Google is blending a algorithmic (i.e., determined by math) AND a directory (i.e., determined by people) model is new." I'm not sure which of them is reading the interview most accurately.
On one hand Norvig's admission is deeply thought provoking - but on another hand isn't it obvious that this would be happening on some level. Just like YouTube hires people to watch videos all day and look for copyrighted content, despite all the big promises of a technology to catch the stuff, it would be insane if Google didn't hire people to look at search results and provide feedback.
I'd love to know what kinds of thoughts this provokes in our readers' minds. See also SearchEngineLand coverage, where this story is put into historical perspective.
Share2 TrackBacks
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/2987
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
working for google as a "web page rater" for over 6 months by now. main task is to rate a search result page, look for spam, ,malware, etc.
search result pages are also subject to comparison.
i think this human interaction is a good thing for google.tons of pages with absolutely no content are rated negatively and are gone from the front page.
also caught some scam-pages, some malware pages, etc.
While any algorithm is awesome and can do alot of stuff, it seems to still be suffering from the lack of human perception and the ability to think "out of the box". I am assuming that's what is needed here for Google due to the fact that there's alot of random chance out there that it can't pick up by looking at it like a human can.
Does this mean that finally Bird On A
Wire my blog will get some rating from the Great Google behind the curtain? Quite frustrating to build it and have no one come because they can't find it!
Algorithm: math + human elements = less spam and no bad sites. Cool :)
Nhick
http://www.itrush.com
This article is great, I put you guys on my blog, hope you will be getting a few 100 + clicks, .By the way if you type in bangkok airport transfer, then i think budapestairportstransfer comes up interesting, http://www.opentopix.com/topic/tech-news/google-pays-people-to-check-the-actual-searches
Tried to comment right when the post was made but the system wasn't working...sorry it's so late!
---------
I used to work for a top-three search engine. Checking search result lists against human judgment is standard practice, both in commercial search and in academic research. Google would be crazy NOT to do this!
The purpose is not to manipulate individual results lists. The number of unique queries is in the hundreds of millions, so it's just not worth it. Instead, people use studies to improve the algorithm and benchmark against competitors.
Studies can be conducted in two ways:
1) Check whether a specific site or sites comes up for a search term. For example, does the official site for the organization Environmental Defense come up if someone searches for "environmental defense"? This is most useful for small collections, such as back issues of an academic journal, where there are a small number of "best" hits that are easy to identify.
2) Rate the results according to how relevant they are to the search term. For example, testers might rate each result on a scale of 1 to 10 for how good a match it is. This is more useful in a huge collection like Internet search, where we could argue for weeks about the "ideal" list of results for any one term.