It does not mean that everybody has to participate. The idea of forcing people to participate contradicts the whole notion of the "Wisdom" that you are seeking to obtain. For example in Predictive Political Markets, its not all gamblers who participate, or all people who follow politics, and neither are its participants widely distributed throughout the country, and its mostly white males.

But those who participate do so voluntarily and its the aggregation of their bets which results in better polling data than Gallup, even though they are not experts.

The same with Amazon reviews, the crowd is those who voluntarily seek to write a review. I am sure these individual people don't come from the same professional backgrounds, and academic experiences, they were not sought out by Amazon.com because they fit some criteria. No, they self-selected themselves individually.

This guy has a simplistic notion of what "Wisdom of the Crowd" means. He is actually taking the term crowd literally to mean that every single person MUST be involved, even if he or she does not want to. Again, he should read the book.

There are two key ways that this problem is addressed on our site: first. suggesting ideas and voting on those ideas are two separate steps in our brainstorming process, and so are not necessarily performed by the same crowds. This decoupling is critical in helping to objectify the value of the crowd generated ideas.

Secondly, we have a voting algorithm takes into account the voting behavior of each individual participant. If they have an unduly biased voting pattern, their vote is worth less than someone who has a proven, egalitarian voting record.

In short, our crowdsourcing framework is designed to address this concern from the get-go.

So while it's true that simple, brute force "wisdom of crowds" has to deal with these issues, we are dealing with systems here, not simple straw polling. As long as the problems can be recognized, they can be addressed, and intelligent systems design used to mitigate the impact.

Like others have said, Muhammad details the requirements for a smart crowd well.

At Intrade (a real money prediction market for those not familiar with us) perhaps 1 person contributes/predicts/trade for every 100 that view the information.

Further, perhaps 5% of our users generate 80% of the predictions.

Where real money is required to back up your opinion/prediction it removes a lot of noise but very occasionally people can use money to try manipulate the aggregated wisdom.

As for whether crowds that satisfy the criteria so defined are smarter then pundits, there is plenty of academic research to support and if we look hard I suppose a bit to detract.

Regards from Dublin,

John

http://www.forrester.com/Groundswell/profile_tool.html

For example, if twenty experts predict next year's inflation rate, then the average of their predictions will be more accurate when they DID NOT talk to each other about their predictions (as compared to when they did). In the latter case, their errors will become more similar because of the discussion and thus their errors are less likely to cancel out. The same logic also applied to voting procedures.

Surowiecki's (2004) book "The Wisdom of Crowds" (and other work) clearly spells out this boundary conditions of "The Wisdom of Crowds". He (and other people in this area) don't simply claim that having a bunch of people do something will be inevitably wise. It's the independence of opinions that's key!

So please stop bashing a strawman and actually criticize "The Wisdom of Crowds" as it actually was proposed.

Similar things could be said about IMDB's user-voted films list. The system can work - though I agree wholeheartedly with the other posters who caution against some users with strong agendas polluting the results. Algorithms need to be a moving target. It's possible that rateyourmusic's accuracy has to do with it not being a very professional site and therefore off the radar of those who would try to game it.

We've just launched an ambitious beta startup at http://www.ranker.com that will also harness the wisdom of crowds, though that feature has not yet been made user-facing.

I mean, the wisdom of the crowd actualy says the opposite of what you find on most of the material published on social networks: small world phenomena, centrality, hubs, power law distribution of connections, etc.

In a live talk I saw Surowiecki give shortly after the book's release, he began with the statement that while ants as a crowd become more "intelligent", humans--as a crowd--become dumber. The unfortunate legacy of his book's misleading title is that those who did not read it have still used it to support a "None of us is as smart as All of us..." approach based on collaboration and consensus that--for the most part--the book demolishes.

Whether it's the proportion of blog commenters vs. blog readers, forum regulars vs. forum lurkers, or content creators vs. content consumers (like in virtual world Second Life), the "actives" are an incredibly devoted subset of the "passives". And someone who's very active on one site may be passive on others depending on how much they're able or willing to contribute. A CHOICE. Again, not a "good" or "bad" thing — many people enjoy watching YouTube videos rather than make them — this is just how it is, and clouding thoughts with an abstract "crowd" isn't helpful when what really matters are individual personalities. Sometimes collaborating, yes, but that's more vivid, peaky grouping (and certainly consistent with some of what Surowiecki suggested).

Not everyone has an interest in being heard, but severe actives often possess a clear proclivity towards self-expression and being rewarded for it — not necessarily in cold cash, but peer recognition and feeling special. Maybe swag. After all, from academia studies to urban art, having your ideas supported is nourishing! An early trickle of reward, even encouragement from the creators or earlier stars may very well snowball and turn a casual hobby into being a major voice on a site like Wikipedia or Amazon's reviews. I've grown to this position multiple times in varied online communities, so through experience, I can relate to how the process develops. And THOSE are the stories I'm most interested in.

As described in ambient "social media" books about reaching out to influential individuals instead of appealing to a faceless mass: power users (hence the name) will continue to wield wide impact over others, such as lesser-ranked currying their favor (like linking) to climb up the social ladder and learning how to ascend in the process.

In other words a utopia

In my work, I've considered and used the four principles as we attempt to pull together the types of collaboratives whcih will allow for the truly creative thinking necessary in changing health care. While aware of the controversy around Surowiecki's work (likely suggesting he's hit on some sensitive nerves), I concur with many of the comments, specifically those of Mohammed, and have found this a useful construct.

Of course it is only one aspect of moving collaborative efforts forward. I've also considered the adaptive/technical leadership model of Ron Heifetz, Otto Scharmer's Theory U, and have used aspects of Group Genius as written by Keith Sawyer. An additional useful construct in the work has been incorporating the elements of Fair Process outlined by Kim in the Harvard Business Review in 1997.

Just my take on looking at the convergences and synergies of different concepts and philosophies, and drawing from them. Of course, attention to the appropriate and attentive use of the concepts is essential to prevent the unintended consequences of misapplication of these concepts.....much more to learn.

In addition to knowing who submitted something, Digg also breaks Rule #2 (Independence) by showing how many diggs something has. This is clearly social bias...Duncan Watts did an amazing study showing how much influence these numbers can have on someone's behavior: http://bokardo.com/archives/social-design-101-aggregate-displays-change-user-behavior/

The idea of wisdom of crowds is that people make decisions independently, and then by aggregation we find out what the crowd thought.

In general, the wisdom of crowds theory isn't as applicable as originally thought...lots of social systems defy the rules in order to drive positive engagement.

I think the interplay between different crowds (i.e., different points of view) has been frequently referred to as politics. Per Mohammed's fine comments (above), when individual viewpoints can be observed without information cascades and the influence of identity, you might be able to glimpse the actual wisdom of the majority at a given point in time. Of course accepting or rejecting that wisdom is a highly individual decision.

I followed prediction markets for the US presidential election and they seemed spot on. There are numerous studies showing that they predicted the results more accurately than any http://www.semanticsincorporated.com/2008/11/90-of-an-obama-win-as-per-prediction-markets.html

Limitations in predictions market are well documented (and include Muhammad's points above), and constrain their practical application to a well-defined number of situation.

Crowdsourcing suffers from the same limitations, which is not a problem, as long as you limit its application correspondingly. The problem occur when you stretch it outside the required constraints and yet present the results as "scientific", i.e. as a good proxy for what the crowd thinks.

That's what professor Vassilis Kostakos's theory ultimately comes down to (or should - I don't know, I haven't read his report). Apps like Digg or Amazon's review are not scientific applications of crowdsourcing, and thus their results should not be seen as precise representation of our collective thinking.

On the one hand, I don't think they present themselves as scientific, so attacking them for that is a little controversy-seeking. On the other hand, they also don't advertise how much they rely on such a small group of reviewers, and they probably should - else they should correct their algorithm further to prevent it. Ultimately, I applaud professor Vassilis Kostakos for his role as a watchdog, and helping improve crowdsourcing applications... in a crowdsourcing kind of way.

I don't think so at all. The problem with most of these sites is that they suffer from information cascades and people participating don't always have 'pure motives.'

From Wikipedia:

There are two key conditions in an information cascade model:

1. Sequential decisions with subsequent actors observing decisions (not information) of previous actors.
2. A limited action space (e.g. an adopt/reject decision).

What happens on Digg is that a lot of stories get Dugg because a particular user submitted them, or particular users Dugg them, or they were from a particular domain, or simply because someone asks you to Digg something. Therefore decisions aren't made with merit in mind (or Suriwiecki's four conditions in mind) and instead popularity rules the day.

I can't remember the name of the site but I recall there was a foreign version of Digg that didn't show you who submitted a story, that's a step up. Similarly Reddit doesn't show you how many votes a story has for the first hour. Things like that try to prevent information cascades.

1. Diversity of opinion [ummmm]
2. Independence [hmmmm]
3. Decentralization [maybe]
4. Aggregation [check, except it aggregates just a handful of sites, like Ars Technica and a few other ones popular with digg's power users]

By the way, I remember hearing somewhere that two’s company, but can’t remember what they call three. It’s right on the tip of my tongue … aww, shucks. I can’t remember.

Nevermind.

Back to your lives, citizens.

Nice summary. For those running UGC sites, it's no secret that a small percentage of the user base creates the vast majority of the content.

At RateItAll (www.rateitall.com), we are trying a number of things to improve the trustworthiness of our content:

- We offer a newsfeed format that populates your review stream with a combination of reviews from your friends, and reviews of topics you are interested in. We give you the option of moving out in degrees of network separation (e.g. "show me reviews of friends of friends of friends") with the idea that networks are self policing. Presumably, people you know in real life won't deliberately steer you the wrong way.

- We provide "compatibility tests" that let you find reviewers who most resemble you in outlook on a category by category basic (e.g. music vs. politics). This was the inspiration for Lunch's similarity network.

- We offer a proprietary "confidence score" that helps determine the best product / item in a certain category. This score factors in a number of results - not just rating. See this dog food ranking for an example: http://www.rateitall.com/t-353-dog-food-brands.aspx (Yelp is also aggressive with automated algos to weed out reviews that look shillish)

- We provide total transparency on a reviewer's rating history. Clicking a user's profile name link shows you immediately all of their ratings activity on the site, without making you dig.

- We provide review by review voting.

- We allow commenting on any review - if you shill, you will likely be called out publicly on it by our regulars.

- We encourage people to login using Facebook Connect (real identity is a major factor when talking about trustworthiness of a review - there's a new site called Oyster.com that ONLY will accept reviews from folks logged in via FB Connect)

- As a horizontal rating site, we try and offer something for everyone. The more data and more participants, the more trustworthiness.

- Similarly, we have tried to remove as many barriers to participation without losing identity - for example, we recently launched a tool that lets you post a review and signup just by sending an email to reviews@rateitall.com - the product / place you are reviewing goes in the subject line, and the rating and review goes in the body.

Transparency. The ability to isolate reviews from people you trust in real life. Trust algorithms. People discovery. Reputation. Broad participation. Real identity. Accessibility.

In my opinion, these are the keys to maximizing the trustworthiness of user generated reviews.

Lawrence Coburn
CEO, RateItAll (www.rateitall.com)

This article said it best, over three years ago:

DIGITAL MAOISM
The Hazards of the New Online Collectivism
By Jaron Lanier

Surowiecki is very clear about the conditions necessary for a wise crowd to prevail and those conditions are:

1. Diversity of opinion
2. Independence
3. Decentralization
4. Aggregation

If your crowd possesses those qualities then it is wise and then it will be better at making decisions under Surowiecki's paradigm. The crowds used in the research (and the crowd in general) doesn't possess those qualities and therefore is an unfit data set. We should be trying to create the ideal crowd before we can obtain superlative results and not try to get good results from any random crowd.