To conduct the research, Kostakos worked with a large sample of online ratings. As MIT's Technology Review reports, the researcher and his team studied hundreds of thousands of items and millions of votes across all three sites. In each and every case, they discovered that small numbers of users accounted for the largest number of ratings. For example, on Amazon, only 5% of active Amazon users ever cast votes on more than 10 products but a small handful of users voted on hundreds of items. Said Kostakos, "if you have two or three people voting 500 times, the results may not be representative of the community overall."

This is hardly the first time that the so-called "wisdom of the crowds" has been called into question. The term, which implies that a diverse collection of individuals makes more accurate decisions and predications than individuals or even experts, has been used in the past to describe how everything from Wikipedia to user-generated news sites like Digg.com offer better services than anything created by a smaller group could do.

Of course, we now know that simply isn't true. For one thing, Wikipedia isn't written and edited by the "crowd" at all. In fact, 1% of Wikipedia users are responsible for half of the site's edits. Even Wikipedia's founder, Jimmy Wales, has been quoted as saying that the site is really written by a community, "a dedicated group of a few hundred volunteers."

And as for Digg.com, a site whose algorithm is constantly tweaked in attempts to democratize the votes of its users, it still remains a place where a handful of power users can make or break getting a news item to the site's front page.

Attempts to Address the Issue

It's not surprising then to discover that, when it comes to review sites, it's again small groups that are in control there too. Some sites, including Amazon, attempt to address this discrepancy by allowing users to vote on the helpfulness of reviews - a much easier process than having to write a review yourself. Also, local business finder and recommendations site Yelp implemented ways for business owners to respond to what they feel are inaccurate reviews by way of an owner comments feature. Unfortunately, despite these efforts, the small groups still remain in control of these so-called "popular opinion" features.

According to the article, another professor at CMU, Niki Kittur, suggested that sites create new tools for transparency. For example, there should be an easy way to see a summary of a user's contributions which would quickly reveal any bias. He also suggested removing overly positive and negative reviews.

Earlier this year, we looked at another user-generated review site which attacked this problem from another angle. Lunch.com, a new Yelp competitor, uses something they call their "Similarity Network" which matches you to other site users who share your interests. That way, instead of looking at a list of reviews which could originate from anyone with an agenda or axe to grind, you're focused on reviews from others like you.

Still, there is yet to be a perfect solution to the problem. Perhaps it's time we give up the idea that the "wisdom of the crowds" was ever a driving force behind any socialized, user-generated anything and realize that, just like in life, there will always be active participants as well as the passive passerbys.

"It's a trap. When you download it runs a validation check to see if you are running a pirated version of photoshop. Which then logs your ip back to Adobe HQ who then mark the ip address in the automated billing system. You will recieve [sic] a fine for $500 in the next 2 to 5 working days. Congratulations" -- luke16

The problem, though, that's not true. The book download is just a PDF file; it doesn't run a version check on Photoshop, it doesn't log your IP address, and SitePoint has no relationship with Adobe. Nonetheless, enough Digg users bought into luke16's active imagination that the story was buried.

"So anyone else in the digg community who might be interested in a full, print-quality Photoshop book -- sorry, you miss out," wrote Tilley. "All because some goose decided to throw around some unsubstantiated claim about the legitimacy of our giveaway. What's worse is that everyone believed him!"

Crowd Rules

SitePoint's experience is an example of herd behavior or groupthink, where the Digg group acted blindly on poor information, without rationally thinking it through. This is a problem with the wisdom of crowds concept: if unchecked, rather than coming to the best conclusion based on the wisdom of the group, a crowd can come to the worst conclusion based on dumbness that spreads from a single bad node.

Last year, we laid out a set of rules to get the most out of a crowd. It might be a good idea to revisit those here:

1. Crowds should operate within constraints. To harness the collective intelligence of crowds, there need to be rules in place to maintain order.
2. Not everything can be democratic. Sometimes a decision needs to be made, and having a core team (or single person) make the ultimate decision can provide the guidance necessary to get things done and prevent crazy ideas and groupthink from wreaking havoc on your product.
3. Crowds must retain their individuality. Encourage your group to disagree, and try not to let any members of the group disproportionately influence the rest.
4. Crowds are better at vetting content than creating it. It is important to note that in most of the above projects, the group merely votes on the final product; they do not actually create it.

Digg's problem lies in the third point -- members were able to quickly spread undue influence on the group via poor information that caused undesired results before that information could be properly vetted by the group for accuracy. Eventually, more reasoned commenters on Digg shot down luke16's paranoid conspiracy theory, but by that time it was too late, the story had already been buried.

Digg probably gets it right far more often than it gets it wrong, but SitePoint's experience is a lesson in the dangers of letting a crowd run wild. Any site that relies on a crowd to organize information should be wary of things like this happening.

When adding a decision to SocialThumbs, authors are asked to input a question (like, "Should I wear red to the prom?"), a motivation (like, "I want to look hot at the prom")," and some background information (like, "My favorite color is red," "My date is color blind," "I have brown hair and green eyes," etc.). The decider then lists pros and cons and offers a score of -5 to +5 for each. Once the decision is public, other users are invited to add additional pro and con reasons, vote on any of the pro/con statements, and leave comments. Based on the overall scoring data, SocialThumbs comes up with a yes or no suggestion.

It strikes us that this site might do well as an application on social networks like Facebook or MySpace where users could call on their friends to help them make decisions.

SocialThumbs is an interesting idea, and while it's not truly a wisdom of the crowds app (we use that term loosely around here), it makes sense to lean on the wisdom of others to help clarify the decision making process. Ultimately, people need to make decisions for themselves, but using SocialThumbs might shed light on aspects of an issue that users hadn't thought of, or offer alternative views that are helpful in their decision making. To paraphrase the Beatles, we get by with a little help from our friends.