The Dirty Little Secret About the "Wisdom of the Crowds" - There is No Crowd
Recent research by Carnegie Mellon University (CMU) professor Vassilis Kostakos pokes a big hole in the prevailing wisdom that the "wisdom of crowds" is a trustworthy force on today's web. His research focused on studying the voting patterns across several sites featuring user-generated reviews including Amazon, IMDb, and BookCrossing. The findings showed that a small group of users accounted for a large number of ratings. In other words, as many have already begun to suspect, small but powerful groups can easily distort what the "crowd" really thinks, leading online reviews to often end up appearing extremely positive or extremely negative.
Small Groups, Big Impact
To conduct the research, Kostakos worked with a large sample of online ratings. As MIT's Technology Review reports, the researcher and his team studied hundreds of thousands of items and millions of votes across all three sites. In each and every case, they discovered that small numbers of users accounted for the largest number of ratings. For example, on Amazon, only 5% of active Amazon users ever cast votes on more than 10 products but a small handful of users voted on hundreds of items. Said Kostakos, "if you have two or three people voting 500 times, the results may not be representative of the community overall."
This is hardly the first time that the so-called "wisdom of the crowds" has been called into question. The term, which implies that a diverse collection of individuals makes more accurate decisions and predications than individuals or even experts, has been used in the past to describe how everything from Wikipedia to user-generated news sites like Digg.com offer better services than anything created by a smaller group could do.
Of course, we now know that simply isn't true. For one thing, Wikipedia isn't written and edited by the "crowd" at all. In fact, 1% of Wikipedia users are responsible for half of the site's edits. Even Wikipedia's founder, Jimmy Wales, has been quoted as saying that the site is really written by a community, "a dedicated group of a few hundred volunteers."
And as for Digg.com, a site whose algorithm is constantly tweaked in attempts to democratize the votes of its users, it still remains a place where a handful of power users can make or break getting a news item to the site's front page.
Attempts to Address the Issue
It's not surprising then to discover that, when it comes to review sites, it's again small groups that are in control there too. Some sites, including Amazon, attempt to address this discrepancy by allowing users to vote on the helpfulness of reviews - a much easier process than having to write a review yourself. Also, local business finder and recommendations site Yelp implemented ways for business owners to respond to what they feel are inaccurate reviews by way of an owner comments feature. Unfortunately, despite these efforts, the small groups still remain in control of these so-called "popular opinion" features.
According to the article, another professor at CMU, Niki Kittur, suggested that sites create new tools for transparency. For example, there should be an easy way to see a summary of a user's contributions which would quickly reveal any bias. He also suggested removing overly positive and negative reviews.
Earlier this year, we looked at another user-generated review site which attacked this problem from another angle. Lunch.com, a new Yelp competitor, uses something they call their "Similarity Network" which matches you to other site users who share your interests. That way, instead of looking at a list of reviews which could originate from anyone with an agenda or axe to grind, you're focused on reviews from others like you.
Still, there is yet to be a perfect solution to the problem. Perhaps it's time we give up the idea that the "wisdom of the crowds" was ever a driving force behind any socialized, user-generated anything and realize that, just like in life, there will always be active participants as well as the passive passerbys.
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
I think your headline is misleading and Vassilis Kostakos should read the book before poking holes.
Surowiecki is very clear about the conditions necessary for a wise crowd to prevail and those conditions are:
1. Diversity of opinion
2. Independence
3. Decentralization
4. Aggregation
If your crowd possesses those qualities then it is wise and then it will be better at making decisions under Surowiecki's paradigm. The crowds used in the research (and the crowd in general) doesn't possess those qualities and therefore is an unfit data set. We should be trying to create the ideal crowd before we can obtain superlative results and not try to get good results from any random crowd.
yep there is no crowd listen to yourself wikipedia and digg is manipulated by their owners itself
http://Evolvhealthy.com
You mean to say, of course, that Wikipedia is edited by a minority of unaccountable zealots
Wisdom of Crowds is a crypto-fascist idea; there is no objective truth, there are no facts, truth is what "the crowd" decides it is. You get these unhealthy echo chambers of "activists" setting the agenda.
This article said it best, over three years ago:
DIGITAL MAOISM
The Hazards of the New Online Collectivism
By Jaron Lanier
There are no small groups without a crowd. This is rudimentary logic. And small groups are only representative of something if they are part of a large crowd. Otherwise, the small group does not actually create any impact. Research conducted by prof. Vassilis Kostakos is valid, but does not truly question the vision of Wisdom of the crowd.
it's all about the niche. It's the selection of the relevant from the crowd of the diverse that really drives wisdom.
Hi Sarah,
Nice summary. For those running UGC sites, it's no secret that a small percentage of the user base creates the vast majority of the content.
At RateItAll (www.rateitall.com), we are trying a number of things to improve the trustworthiness of our content:
- We offer a newsfeed format that populates your review stream with a combination of reviews from your friends, and reviews of topics you are interested in. We give you the option of moving out in degrees of network separation (e.g. "show me reviews of friends of friends of friends") with the idea that networks are self policing. Presumably, people you know in real life won't deliberately steer you the wrong way.
- We provide "compatibility tests" that let you find reviewers who most resemble you in outlook on a category by category basic (e.g. music vs. politics). This was the inspiration for Lunch's similarity network.
- We offer a proprietary "confidence score" that helps determine the best product / item in a certain category. This score factors in a number of results - not just rating. See this dog food ranking for an example: http://www.rateitall.com/t-353-dog-food-brands.aspx (Yelp is also aggressive with automated algos to weed out reviews that look shillish)
- We provide total transparency on a reviewer's rating history. Clicking a user's profile name link shows you immediately all of their ratings activity on the site, without making you dig.
- We provide review by review voting.
- We allow commenting on any review - if you shill, you will likely be called out publicly on it by our regulars.
- We encourage people to login using Facebook Connect (real identity is a major factor when talking about trustworthiness of a review - there's a new site called Oyster.com that ONLY will accept reviews from folks logged in via FB Connect)
- As a horizontal rating site, we try and offer something for everyone. The more data and more participants, the more trustworthiness.
- Similarly, we have tried to remove as many barriers to participation without losing identity - for example, we recently launched a tool that lets you post a review and signup just by sending an email to reviews@rateitall.com - the product / place you are reviewing goes in the subject line, and the rating and review goes in the body.
Transparency. The ability to isolate reviews from people you trust in real life. Trust algorithms. People discovery. Reputation. Broad participation. Real identity. Accessibility.
In my opinion, these are the keys to maximizing the trustworthiness of user generated reviews.
Lawrence Coburn
CEO, RateItAll (www.rateitall.com)
Sounds like there is a vast conspiracy going on here. I'm sure of it. A small group of impotent, brainless little twit-faced pee-ons are trying to suck the control and credit out of the hands of those greedy, power-hungry, know-it-all experts. Oh yeah. Treachery is afoot.
By the way, I remember hearing somewhere that two’s company, but can’t remember what they call three. It’s right on the tip of my tongue … aww, shucks. I can’t remember.
Nevermind.
Back to your lives, citizens.
Agree with the previous commenter. The situations described in the research do not relate to a crowd as defined in Surowiecki's book.
Re "Of course, we now know that simply isn't true. For one thing, Wikipedia isn't written and edited by the "crowd" at all" see http://www.aaronsw.com/weblog/whowriteswikipedia
Muhammad, do you think Digg complies with those 4 points in the book?
1. Diversity of opinion [ummmm]
2. Independence [hmmmm]
3. Decentralization [maybe]
4. Aggregation [check, except it aggregates just a handful of sites, like Ars Technica and a few other ones popular with digg's power users]
Hi Richard,
I don't think so at all. The problem with most of these sites is that they suffer from information cascades and people participating don't always have 'pure motives.'
From Wikipedia:
There are two key conditions in an information cascade model:
1. Sequential decisions with subsequent actors observing decisions (not information) of previous actors.
2. A limited action space (e.g. an adopt/reject decision).
What happens on Digg is that a lot of stories get Dugg because a particular user submitted them, or particular users Dugg them, or they were from a particular domain, or simply because someone asks you to Digg something. Therefore decisions aren't made with merit in mind (or Suriwiecki's four conditions in mind) and instead popularity rules the day.
I can't remember the name of the site but I recall there was a foreign version of Digg that didn't show you who submitted a story, that's a step up. Similarly Reddit doesn't show you how many votes a story has for the first hour. Things like that try to prevent information cascades.
Great points Muhammad. That would indeed be interesting, if Digg hid the submitter name!
An interesting application of crowdsourcing are decision markets, see http://en.wikipedia.org/wiki/Prediction_market As a sidenote, it's quite ironical that wikipedia highlights the shortcomings of those markets in their entry - but in their defense they also include the criticism on their own bias in their entry http://en.wikipedia.org/wiki/Wikipedia
I followed prediction markets for the US presidential election and they seemed spot on. There are numerous studies showing that they predicted the results more accurately than any http://www.semanticsincorporated.com/2008/11/90-of-an-obama-win-as-per-prediction-markets.html
Limitations in predictions market are well documented (and include Muhammad's points above), and constrain their practical application to a well-defined number of situation.
Crowdsourcing suffers from the same limitations, which is not a problem, as long as you limit its application correspondingly. The problem occur when you stretch it outside the required constraints and yet present the results as "scientific", i.e. as a good proxy for what the crowd thinks.
That's what professor Vassilis Kostakos's theory ultimately comes down to (or should - I don't know, I haven't read his report). Apps like Digg or Amazon's review are not scientific applications of crowdsourcing, and thus their results should not be seen as precise representation of our collective thinking.
On the one hand, I don't think they present themselves as scientific, so attacking them for that is a little controversy-seeking. On the other hand, they also don't advertise how much they rely on such a small group of reviewers, and they probably should - else they should correct their algorithm further to prevent it. Ultimately, I applaud professor Vassilis Kostakos for his role as a watchdog, and helping improve crowdsourcing applications... in a crowdsourcing kind of way.
In business you have the 20/80 rule.
In social website possibly there is a 1/99 rule or even less.
Meaning a very small number of people actually generate content. From this small group a large part has a different agenda as the content they generate has a self-benefit purpose.
Hey I found this website called HTcity.desimartini.com , it has a good collection of Page 3 party pics.
While I think it always important to question popular sentiment...I also think a crowd is defined primarily by its members participation. If you aren't contributing to some group, then your viewpoint may or may not be represented by others in that group.
I think the interplay between different crowds (i.e., different points of view) has been frequently referred to as politics. Per Mohammed's fine comments (above), when individual viewpoints can be observed without information cascades and the influence of identity, you might be able to glimpse the actual wisdom of the majority at a given point in time. Of course accepting or rejecting that wisdom is a highly individual decision.
Richard, Muhammad is right. Suroweicki is adamant that the four rules be followed for the wisdom of crowds to take effect.
In addition to knowing who submitted something, Digg also breaks Rule #2 (Independence) by showing how many diggs something has. This is clearly social bias...Duncan Watts did an amazing study showing how much influence these numbers can have on someone's behavior: http://bokardo.com/archives/social-design-101-aggregate-displays-change-user-behavior/
The idea of wisdom of crowds is that people make decisions independently, and then by aggregation we find out what the crowd thought.
In general, the wisdom of crowds theory isn't as applicable as originally thought...lots of social systems defy the rules in order to drive positive engagement.
An interesting array of comments, and concerns, which are fascinating in their own right. I've been cognizant of Surowiecki's work, and his four principles for assuring the ability to bring out the "wisdom of crowds" in addressing complex, highly charged topics, while avoiding the "dumbness" and group think so commonly seen. (The Bay of Pigs analogy is a useful one).
In my work, I've considered and used the four principles as we attempt to pull together the types of collaboratives whcih will allow for the truly creative thinking necessary in changing health care. While aware of the controversy around Surowiecki's work (likely suggesting he's hit on some sensitive nerves), I concur with many of the comments, specifically those of Mohammed, and have found this a useful construct.
Of course it is only one aspect of moving collaborative efforts forward. I've also considered the adaptive/technical leadership model of Ron Heifetz, Otto Scharmer's Theory U, and have used aspects of Group Genius as written by Keith Sawyer. An additional useful construct in the work has been incorporating the elements of Fair Process outlined by Kim in the Harvard Business Review in 1997.
Just my take on looking at the convergences and synergies of different concepts and philosophies, and drawing from them. Of course, attention to the appropriate and attentive use of the concepts is essential to prevent the unintended consequences of misapplication of these concepts.....much more to learn.
"Suroweicki is adamant that the four rules be followed for the wisdom of crowds to take effect."
In other words a utopia
What I'd like to see is non-fakeable metrics on ecommerce sites: return rates or reorder rates (as appropriate), for example. Or for apps, how many times users open the app per day/week or whatever.
This is not just about TWoC, but human behavior: a small amount of people have, are, and will continue to contribute disproportionately. This is neither a "good" or "bad" thing of itself; simply observed behavior. And it's nothing new, especially when we consider eccentrics (meant in the most complimentary sense) who possess incredibly intense amounts of focused knowledge, and thanks to the Internet, are able to easier share their encyclopedic bearings on retrogaming, weird stuff happening in Asia, or even the quintessential archetype of the sci-fi geek.
Whether it's the proportion of blog commenters vs. blog readers, forum regulars vs. forum lurkers, or content creators vs. content consumers (like in virtual world Second Life), the "actives" are an incredibly devoted subset of the "passives". And someone who's very active on one site may be passive on others depending on how much they're able or willing to contribute. A CHOICE. Again, not a "good" or "bad" thing — many people enjoy watching YouTube videos rather than make them — this is just how it is, and clouding thoughts with an abstract "crowd" isn't helpful when what really matters are individual personalities. Sometimes collaborating, yes, but that's more vivid, peaky grouping (and certainly consistent with some of what Surowiecki suggested).
Not everyone has an interest in being heard, but severe actives often possess a clear proclivity towards self-expression and being rewarded for it — not necessarily in cold cash, but peer recognition and feeling special. Maybe swag. After all, from academia studies to urban art, having your ideas supported is nourishing! An early trickle of reward, even encouragement from the creators or earlier stars may very well snowball and turn a casual hobby into being a major voice on a site like Wikipedia or Amazon's reviews. I've grown to this position multiple times in varied online communities, so through experience, I can relate to how the process develops. And THOSE are the stories I'm most interested in.
As described in ambient "social media" books about reaching out to influential individuals instead of appealing to a faceless mass: power users (hence the name) will continue to wield wide impact over others, such as lesser-ranked currying their favor (like linking) to climb up the social ladder and learning how to ascend in the process.
Not to mention the fact that the "wisdom" of crowds does not bring out any objective truth, just a collective subjectivism.
Joshua Porter's comment ++
In a live talk I saw Surowiecki give shortly after the book's release, he began with the statement that while ants as a crowd become more "intelligent", humans--as a crowd--become dumber. The unfortunate legacy of his book's misleading title is that those who did not read it have still used it to support a "None of us is as smart as All of us..." approach based on collaboration and consensus that--for the most part--the book demolishes.
That's why I dont read books on technology and society trends written by journalists... they are all abou selling a good story, and miss out on the substance all to easily. Go for Duncan, Barabasi, Tapscott, people who actually have experience and research to back their opinions.
I mean, the wisdom of the crowd actualy says the opposite of what you find on most of the material published on social networks: small world phenomena, centrality, hubs, power law distribution of connections, etc.
You miss the other key point: the Crowd isn't very clever http://www.itskeptic.org/folly-crowd
The wisdom of crowds does work. The best example I have seen on the web is http://www.rateyourmusic.com - it's not a site with any bells and whistles, but if you are a music fan go search for your favorite band, look at the list of albums and "crowd ratings" that come up, and see if the crowdsourced ratings match what you (or the critics at large) think are the best, and worst, records by a given band. I bet you'll agree.
Similar things could be said about IMDB's user-voted films list. The system can work - though I agree wholeheartedly with the other posters who caution against some users with strong agendas polluting the results. Algorithms need to be a moving target. It's possible that rateyourmusic's accuracy has to do with it not being a very professional site and therefore off the radar of those who would try to game it.
We've just launched an ambitious beta startup at http://www.ranker.com that will also harness the wisdom of crowds, though that feature has not yet been made user-facing.
The Guardian (uk newspaper) had online an article about people claiming their editing was undone on wikipedia, and the comments posted in response included eyewitnesses having their stuff deleted, and apparently you get higher ranked the more edits you make, many people compete, and the quickest edit is 'delete' so to get points fast, these types just delete every edit as soon as they find it. :(
Interesting article. However, I'd like to point out one fundamental necessary condition for "The Wisdom of Crowds" that is often overlooked (also in this article): Crowds are only wise when the members come up with their individual opinions INDIVIDUALLY, that is, when they are NOT influenced by what other people already said, wrote etc.
For example, if twenty experts predict next year's inflation rate, then the average of their predictions will be more accurate when they DID NOT talk to each other about their predictions (as compared to when they did). In the latter case, their errors will become more similar because of the discussion and thus their errors are less likely to cancel out. The same logic also applied to voting procedures.
Surowiecki's (2004) book "The Wisdom of Crowds" (and other work) clearly spells out this boundary conditions of "The Wisdom of Crowds". He (and other people in this area) don't simply claim that having a bunch of people do something will be inevitably wise. It's the independence of opinions that's key!
So please stop bashing a strawman and actually criticize "The Wisdom of Crowds" as it actually was proposed.
Relying on crowd opinions can be helpful, up to a certain point. Sites like Yelp are usually pretty accurate when it comes to group consensus about local businesses. Voting on the helpfulness of a particular comment can help weed out (or at least minimize) reviews that don't contribute to the community in a significant way.
Yep, agree with all those who state that the research does not fit with Surowiecki's definition of the crowd. However, the research is interesting if linked to ideas of unrepresentative or illiberal democracy, as posited by Fareed Zakaria that suggests small interest groups can hijack democratic systems.
Nobody said to trust cloud wisdom. Because that is like 'majority rules' or 'committee decision'. And every human decision can be wrong in the first place or cango wrong. Cloud decision making is just a way of offloading decision making factors and the task of coming up with supporting factors and points unloaded to somebody else so you could review what other people think about the issue and take your own sweet decision. This is how large companies keep a pulse on public opinion. Like google building hype around google wave. Instead of building it as separate app they could have just built it as another tab in a google personal dashboard or nerve center - the tabs would consist of email, reader, notebook, personal wiki, bookmarks and social media dashboard (google wave), igoogle and any other labs gadget you could think of (like fastflip) and a real time ticker of data (like from reinvogarate/snoop or from eyebrowse ticker)
This doesn't surprise me at all. It's a lot like focus groups. Seems like a great way to get feedback until you realize that a) the stats are skewed by the fact that people who are willing to participate in focus groups are different from those who aren't, and b) there are always one or two people in every focus group that influence the others. Crowd sourcing can't be too totally different. In every crowd, there are always voices that are louder and hold more sway. That's how leadership works, for better or worse.
A small group of people can affect the opinions of a much larger crowd, simply by being overly assertive in their views. Your example of the reviews at Amazon shows that polls need not necessarily reflect the thinking of the people, because they may be distorted by a small group.
In this context, Forrester has been very insightful on this for some time with their Social Technographics ladder.
http://www.forrester.com/Groundswell/profile_tool.html
I'd upvote your comment but this site doesn't allow that.
Great discussion.
Like others have said, Muhammad details the requirements for a smart crowd well.
At Intrade (a real money prediction market for those not familiar with us) perhaps 1 person contributes/predicts/trade for every 100 that view the information.
Further, perhaps 5% of our users generate 80% of the predictions.
Where real money is required to back up your opinion/prediction it removes a lot of noise but very occasionally people can use money to try manipulate the aggregated wisdom.
As for whether crowds that satisfy the criteria so defined are smarter then pundits, there is plenty of academic research to support and if we look hard I suppose a bit to detract.
Regards from Dublin,
John
Crowds always follow. So it all goes to individuals that influence crowds.
Muhammad's surely right to defend the original criteria, but isn't the more interesting problem, as Relenz says, that crowds are never, or at least very rarely, independent and as such their wisdom is generally dubious?
As the creator of a crowdsourcing site for brainstorming, BountyStorms.com, I would have to agree with the general gist of the article, while disagreeing that the wisdom of crowds must always be compromised by the influence of a few key participants.
There are two key ways that this problem is addressed on our site: first. suggesting ideas and voting on those ideas are two separate steps in our brainstorming process, and so are not necessarily performed by the same crowds. This decoupling is critical in helping to objectify the value of the crowd generated ideas.
Secondly, we have a voting algorithm takes into account the voting behavior of each individual participant. If they have an unduly biased voting pattern, their vote is worth less than someone who has a proven, egalitarian voting record.
In short, our crowdsourcing framework is designed to address this concern from the get-go.
So while it's true that simple, brute force "wisdom of crowds" has to deal with these issues, we are dealing with systems here, not simple straw polling. As long as the problems can be recognized, they can be addressed, and intelligent systems design used to mitigate the impact.
Wisdom of the minority? I don't think so. It's unavoidable logic if you think it through. Wisdom of the crowds versus wisdom of the powerful. Get a grip.
There are better review sites out there like ezippy.com that lets people do more.
I also firmly believe most contributions to Wikipedia come from a vast array of people: the crowd. This is why it is so interesting that only 2% of this crowd, the editors, are allowed to vote. Do you see the patern? Those who do 99% of the job, for free, those who have the knowledge, are not only not-recognized as such by Jimbo Wales (what would it cost to just thank them?), but they are denied any real ownership on their work and then denied any right to empower (vote for) any other "simple people" while an elite, a high priesthood, is given all authority. Since these few hundred editors are supposed to be independent (although they may depend on each other's votes) I guess they cannot be lobbyed or threatened into any form for censorship, right? Although the worst form of censorship certainly is the guy who deletes or utterly destroys the competent work of knowlegeable contibutors for counter-productive "editorial" reasons. Sounds a lot to me like the mainstream media. Take a look at this video, with an interesting comment from Jimbo, on how the elites' approach clashes with the real power of the wisdom of crowds: http://www.youtube.com/watch?v=Z82B1zsvyZU&feature=related
I don't get how this research contradicts the Wisdom of the Crowd. I get the feeling that the research did not read the book and does not understand what is meant by the wisdom of the crowd.
It does not mean that everybody has to participate. The idea of forcing people to participate contradicts the whole notion of the "Wisdom" that you are seeking to obtain. For example in Predictive Political Markets, its not all gamblers who participate, or all people who follow politics, and neither are its participants widely distributed throughout the country, and its mostly white males.
But those who participate do so voluntarily and its the aggregation of their bets which results in better polling data than Gallup, even though they are not experts.
The same with Amazon reviews, the crowd is those who voluntarily seek to write a review. I am sure these individual people don't come from the same professional backgrounds, and academic experiences, they were not sought out by Amazon.com because they fit some criteria. No, they self-selected themselves individually.
This guy has a simplistic notion of what "Wisdom of the Crowd" means. He is actually taking the term crowd literally to mean that every single person MUST be involved, even if he or she does not want to. Again, he should read the book.