Sometimes Crowds Aren't That Wise
Last week, computer book publisher SitePoint relayed a story about recent experiences with Digg that demonstrates that the Digg system is far from perfect. We've written recently on ReadWriteWeb about the decline and fall of quality on Digg, but SitePoint's anecdote demonstrates that sometimes the wisdom of crowds approach is, well, kind of dumb. Now is probably a good time to revisit the rules for harnessing the wisdom of the crowds we published on this blog a year ago.
SitePoint Marketing Manager Shayne Tilley talked about the company's efforts to promote a recent book giveaway via Digg on an SP blog. Within an hour after the promotion went live it had been dugg 30 times, but then, just as quickly, it was buried. Was it because SitePoint had submitted their own content to Digg, something that Digg users generally frown upon? No, SitePoint hadn't done that, they just put a "Digg This" button on the campaign page. The reason for the bury was likely this comment, according to SitePoint, who noticed the bury come down shortly after the comment was posted:
"It's a trap. When you download it runs a validation check to see if you are running a pirated version of photoshop. Which then logs your ip back to Adobe HQ who then mark the ip address in the automated billing system. You will recieve [sic] a fine for $500 in the next 2 to 5 working days. Congratulations" -- luke16
The problem, though, that's not true. The book download is just a PDF file; it doesn't run a version check on Photoshop, it doesn't log your IP address, and SitePoint has no relationship with Adobe. Nonetheless, enough Digg users bought into luke16's active imagination that the story was buried.
"So anyone else in the digg community who might be interested in a full, print-quality Photoshop book -- sorry, you miss out," wrote Tilley. "All because some goose decided to throw around some unsubstantiated claim about the legitimacy of our giveaway. What's worse is that everyone believed him!"
Crowd Rules
SitePoint's experience is an example of herd behavior or groupthink, where the Digg group acted blindly on poor information, without rationally thinking it through. This is a problem with the wisdom of crowds concept: if unchecked, rather than coming to the best conclusion based on the wisdom of the group, a crowd can come to the worst conclusion based on dumbness that spreads from a single bad node.
Last year, we laid out a set of rules to get the most out of a crowd. It might be a good idea to revisit those here:
- Crowds should operate within constraints. To harness the collective intelligence of crowds, there need to be rules in place to maintain order.
- Not everything can be democratic. Sometimes a decision needs to be made, and having a core team (or single person) make the ultimate decision can provide the guidance necessary to get things done and prevent crazy ideas and groupthink from wreaking havoc on your product.
- Crowds must retain their individuality. Encourage your group to disagree, and try not to let any members of the group disproportionately influence the rest.
- Crowds are better at vetting content than creating it. It is important to note that in most of the above projects, the group merely votes on the final product; they do not actually create it.
Digg's problem lies in the third point -- members were able to quickly spread undue influence on the group via poor information that caused undesired results before that information could be properly vetted by the group for accuracy. Eventually, more reasoned commenters on Digg shot down luke16's paranoid conspiracy theory, but by that time it was too late, the story had already been buried.
Digg probably gets it right far more often than it gets it wrong, but SitePoint's experience is a lesson in the dangers of letting a crowd run wild. Any site that relies on a crowd to organize information should be wary of things like this happening.
ShareRelated Entries
1 TrackBacks
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/4054
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
No.3 can be promoted when users trust only their 'friends' in the community and thus not all recommendations are promoted, just the ones that emerge from your circle.
In Israel we have a very popular news site, YNet, which allows talkbacks (comments) on all articles. These talkbacks are so important and influential that there are more than a few 'professional' talkbackers and many times a fake discussion is created.
Trust the crowd and that's what you get :)
I can't believe they refused an offer that I can't refuse ;)
Interestingly, while the Digg crowd got it wrong, the larger internet crowd is correcting the mis-information. Which goes to say don't stay in one small crowd, feed among other crowds.
Of course, I have come to doubt the wisdom of crowds just watching the political campaign. Crowds are quite susceptible to rumors and conspiracy theories.
My impression is not that the idea is that every crowd is wise, but that if you take the aggregate of decisions made by people acting in their own self-interest, you can get good information.
I think the key is that the people you watch need to have something at stake, something probably more substantial than a mouse click. Investors, for example, are generally very self-correcting. In this case, luke16 didn't lose anything for his ignorance, he already had the book, and likely knew better.
I don't really think the Digg system in general captures much besides popularity (which isn't necessarily bad) and a few concerted efforts to reach the front page.
Nice article, Josh!
I think Digg is not a great example of "Wisdom of Crowds", however, for several reasons:
1. Heavily targeted for gaming: the subversive comment above is an example of negative gaming - voting something down / burying it - but it's undue influence nevertheless.
2. Strong herd mentality, as you pointed out above
3. Voting format is explicit, which is more susceptible to undesirable behavior, than implicit, such as del.icio.us bookmarking, which is much harder to subvert.
4. Control: Thee site is not really democratic, the digg-lords periodically exercise explicit control rather than letting the crowd decide.
I've documented the issues with WoC and compared the positive and negative points of several engines, including digg, del.icio.us and Google, in this (old) post on my blog:
http://blog.softwareabstractions.com/the_software_abstractions/2007/01/search_and_the_.html
Dugg! Wouldn't it be ironic if this article got buried for being critical of Digg? [It would just prove the point!]
I was so happy to read this headline, and the conclusions. You are the first person to address the issue of the downside of herd mentality, AND it needed to be addressed.
Herd mentality can always move a product, or spread a rumor. It can tell if something tastes good, or feels good or bad. But regarding more subtle things, groups are less useful. I have to refer to something I was told long ago:
Most people seek to belong like the drowning move towards life boats. To choose to unravel and follow your own path, requires a strength most do not possess.
But it is this strength that creates shifts and innovation. Someone needs to develop a social site that gathers the input of individuals who prefer to not trail along their masses of "friends". This might be a place where we find another type of post or tweet.
The thing is collective intelligence is, as someone said above, best captured implicitly. By aggregating individual behavior over a period of time, you will find that the best stuff bubbles to the top (one reason del.icio.us works).
Digg has another problem though, which is why I rarely visit. The quality of discussion and what people like is very poor, which is a reflection of the people on the site.
people, crowds, mobs, ascending degrees of insanity
A crowd's wisdom is only as great as those in the crowd.
Digg's audience hasn't ever represented much collective intelligence - its in the nature of the application.
If I look at an article on a new search algorithm, and a 12 year old kid gives it a thumbs up because it looks neat, and a comp sci professor thumbs down because of a technical constraint; is it really a 50-50 weight?
Exactly correct.
Once advertising revenues are involved, people try to game the system(s).
Digg stopped working because SEO people have it so heavily gamed. There is a real chance some competitor intentionally put in that comment just to tank the article.
If you're a large site and not paying a tithe to marketing people, you will get hammered there. If you get hammered a few times, you get on the autobury list.
Getting to the top of Digg has very little to do with content.
There are actually specific, tested rules that have been laid out by James Surowiecki, Larry Lessig, Dave Weinberger and others for how these systems should work.
Your general guidelines are a nice summary, but incomplete. Anyone building a system that relies on community engagement should track their rules down.
Developing a system with the proper checks and balances just takes a bit of research. Many fatal flaws have been adequately tested (not that new ones won't pop up -- but that's impossible to predict).
thanks for the blindingly obvious insight! i hope you are not getting paid to write this stuff! its like warning mail users that they might receive spam if they use email... or search engine users that not all results will be relevant, or ...
For a perfect example of how the wisdom of the crowd isn't always that wise, check this out: http://www.nowpublic.com/tech-biz/fake-hookers-fake-credit-cards-fake-13yr-olds-marketing
great points - one classic problem with wisdom of crowds (particularly when it is serial and not simultaneous) is the overimpact of one negative review (and rarely the opposite)
Good point, Josh. I would recommend you check out Infotopia which is a book about how technology enables decision making. I actually reviewed it here: http://blurringborders.com/2008/05/23/book-review-infotopia-by-cass-sunstein/
but it is worth a read in its entirety.
Cheers....the herd mentality is usually wrong.
Once a rumor starts it is just about impossible to end it.
Live From Las Vegas
The Masked Millionaire
what is interesting about your article is how you fail to recognize the level of paranoia in the general public. every day we find articles detailing the latest attacks of the RIAA, the MPAA, our government and others on citizens for 'suspicious activities.' how could it be surprising that a response that squashed your marketing effort would fall into this line of thinking.
whether you feel that you are a part of the problem or completely outside out of it, unfortunately none of us in the media, the government, the entertainment industry are exempt from this terror and suspicion on the part of the public. you can thank sony for infecting its customers with viruses for that attitude amongst many other companies.
but just as interesting is the uniformity of response within your own comments section. has the hilarious irony of similar thought from all your commenters been lost on you? i hope not; i've been laughing pretty hard at the 'crowd mentality' exemplified here.
which, of course, is what you decry in digg and it's peered ilk. but isn't that what we do? point out our own flaws in others as we fail to meet those standards ourselves?
cheers,
pol rosenthal
Found this article which made me smile on same subject:
http://pilchardboy.tumblr.com/post/35696288/democracy-web-2-0-why-they-both-fall-short
Buried
I think you've witnessed that "crowds" tend to be governed by fear or by bullies.
Digg tends to highlight the fact that the majority are poorly educated and have preconceived ideas. Digg likes to shout down any reasoned arguments and uses dumb reasoning.
You need to target your audience better rather than feed the Digg rabble.
Call me a social networking snob but I prefer to hang-out at delicious and only visit digg when I feel like subjecting myself to the latest "America's Funniest Home Videos" style internet clip or another pedantic political rant. (side note: Digg reaks of irony with its anti-Fox news message and its own propaganda machine of anti-Bush / Clinton diatribes)
anyone who understands the rules under which crowd wisdom operates will know that one member who influences the others reduces the wisdom of the crowd. For a crowd to be truly smart the decisions of each member of the crowd must be made independently.
A herd is not smart but a crowd or group of independent people is....
http://www.sitepoint.com/forums/showthread.php?t=133501
What goes around - comes around.
After being a member on sitepoint for several years and posting news to their forums without any problems - one public complaint about a rude moderator who stole our breaking news posts twice - got a ban.
Life is a circle - what you do to others eventually returns to make you that same victim
I think the idea that crowdsourcing produces better quality or more diversity of ideas than editorial control is missing the point. Both approaches are necessary -- one to bring expertise and careful, researched analysis to the table, the other to correct the biases and ideological corrosion that experts sometimes tend to exhibit.
Crowds are what they are called... Crowds...
The Isaac Isimov create a character, that used the crowd moviments do predict the future and with it's vision of the future, he creates the 2 fundations to save the empire... but I do not think it's possibile to predict a social moviment.
It was a basic example of what WOC guys call information cascades. I thought the article was a bit simplistic, and it's interesting that most of the comments indicate that people don't understand the implications of the wisdom of a crowd. When they complain that one user can have undue influence, they erroneously isolate that instance is bad. Yes, it's a bad thing if the audience is small and you jump into the crowd at the wrong time, but it should cancel itself out over time if the crowd is diverse. (Maybe Digg fails because it emphasizes recent popularity, then its users/readers move on without the time/crowd diversity corrections) Just as stock markets have many bullish market analysts publicly pushing stocks one way, and lots of people follow them, the markets typically have bears pushing the other way, with their own set of herds following them. And then lots of others acting independently. The stock market equilibrium exists (as the theory goes, being as close to the perfect pricing mechanism as you can get) when you have independence on the aggregate. Herd mentalities are fine, as long as they have the opportunity to exist in both directions with a mix of independent thinkers. Surowiecki talks about how crowds with smart, average and dumb people are still better than only smart people. It's diversity that makes the difference.
A crowd is only as smart as it's loudest member. Which on Reddit, Slashdot, Mixx and Propeller is someone with at least average intelligence.