If you're even peripherally involved in the social news space you are
probably familiar with the rather rocky relationship that Digg has
with its core community. Fueled partly by a need to counter false
accusations from disgruntled community members who claim that Digg is
rigged (i.e. that a core group of users decide what content is promoted), partly by the desire to encourage non-core members to participate
more passionately, and partly by a need to affect a level of diversity and
equality that would appear promising to potential acquirers, Digg has
changed its algorithm again and again to artificially favor certain
categories over others (i.e. world news and politics over technology)
and to favor relatively new users over long-time, active users.
This is a guest post by Muhammad Saleem, a social media consultant and a top-ranked community member on multiple social news sites.
Let me assure you that this is not just another rant about how top users aren't being treated fairly. This post isn't about top users or new users, it is purely about the quality of the content on the front page of Digg, and the causes for the decline in the quality of that content. Chief among those causes is the lack of transparency and the imbalance in the algorithm that favors certain users over others and ultimately results in diversity, but poorer (if not poor) content. To understand the change and the effects of the change I analyzed the available data on different users' submitting habits, the Digg algorithm's promotion habits, and the reaction of the community to the content that is ultimately promoted.
The metrics used to gauge content popularity are quite straightforward: the absolute number of Diggs per story and the absolute number of comments per story. The Diggs determine how many other community members - the ones that don't vote to make the story popular but find it worthwhile once it is promoted - like a story, and the comments determine how much engagement and conversation apart from Digging, the story generates. The data used here is for the 20 most prolific users but when I tested for the top 50 the results were similar.
Let's first look at the user rankings based purely in terms of the number of stories promoted in a 30 day period. The users are ranked so that the person with the most number of stories promoted appears first and the one with the least number of stories promoted appears last, the idea being that the person with the most quality content gets the most attention, or more appropriately, that person's content gets the most attention (the most basic principle behind all of Digg and the content promotion algorithm). The ranking is as follows:
Now let's look at the same rankings (i.e., users are ranked by the number of stories promoted), but look at how many Diggs an average story from those users gets once it is promoted. If the algorithm works well in determining quality and is not flawed or artificially helping some users over others, the graph should be exactly the same as above and the people with the most stories promoted are also the people that get the most Diggs per story (because the algorithm is only promoting the best of the best and nothing else).
As you can see, the graph is not the same. What you see on the other hand is that there are some users whose stories are promoted more frequently even though they don't perform as well, while there are other users whose stories consistently out perform but aren't promoted as frequently. What this means is that some users have an easier time getting their content promoted (for whatever reason) but once the content does get promoted, it largely falls flat on the front page.
Now let's look at how the promoted content engages the community in conversation. Again the users are ranked by the number of stories they got promoted (as in the first graph).
Again you can see that some users have an easier time getting their content promoted to the front page but once the content gets there, people aren't really that interested in talking about it. Others, however, have content that everyone has something to say about (even though the algorithm won't let them get stories on the front page as often)
And finally, let's look at the number of stories promoted and the number of Diggs received per story (average) on the same chart (note I had to multiply the promotion data by 10 so the graph would be visible compared to the Diggs per story):
As you can see there is a huge disparity in the number of stories promoted and how viral those stories go. Ultimately, what's happening is that people whose stories almost always get 1200+ Diggs are getting throttled (for whatever reason) whereas people whose stories are routinely getting under 1000 Diggs once promoted, are being favored. It doesn't matter if we're taking about top users or not, because the end result is that lower quality content is promoted to the front page more often than content that performs better after promotion. Case in point, why is it that user "zaibatsu" - whose average Digg-per-story number out paces any Digg user in history (1775 Diggs per story) - gets on average only one story promoted per day while users getting less than 700 Diggs per story get promoted multiple times per day?
Note: This isn't meant as an attack on any user and his or her submissions. All the users mentioned and displayed on the graphs are my friends and I respect their submissions. This post is just meant to point out the flaws of Digg's content promotion algorithm.
Listed below are links to blogs that reference this entry: The Decline and Fall of Quality on Digg.
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/3875
Comments
Subscribe to comments for this post OR Subscribe to comments for all Read/WriteWeb posts
So in a perfect world all the front page stories would be dugg on average around the same number of times?
I think it is all relative and thats what makes the Digg.com front page interesting. If every article promoted was so thought provoking that I felt it was my obligation to digg it then when would I get to see those Motivational Posters, CollegeHumor parodies or irreverent pics that Digg user Pizzler (one of the lowest on this list as far as average diggs per stories goes) submits?
Otherwise you asking Digg to be a niche news site that demands we arouse action for or against the subject of the submission. I like the funny, bland, silly posts as much as the Wired assignments and BBC news stories.
Posted by: MikeonTV | May 1, 2008 6:43 PMWow. This article is full of some amazing stats that clearly took due diligence to compile. As an active digg user who loves hearing other people's theories on the site, it's refreshing that Mu went many steps further than conjecture.
However, I think there's one piece of the puzzle that the article might be skipping over. When articles get promoted to the front page with relatively high numbers of diggs, many a whimsical user is likely to click on that story by virtue of the high number. In other words, it's definitely bush league that top digg users now require such exorbitant numbers of diggs to the hit the FP, but once they arrive there, from a sheer numbers standpoint, it's as though they have a "head start."
Does that make sense? I don't know.
Either way, this article was amazing. Filled with great insight and clarity!
Posted by: BuzzDiggity | May 1, 2008 6:50 PMWanted to clear 2 things up.
1. When talking about "quality" on Digg I mean things that the community would like and would be popular. Quality on Digg is obviously not the same as quality on newstrust.net.
2. The point of the article is not that some users shouldn't be promoted, not at all. The point is that no user should be throttled back. Let everyone play the game.
Posted by: Muhammad Saleem | May 1, 2008 7:29 PMHey Muhammad, I was thinking ... it would also be interesting to compare promotions vs. submissions. I.e., if User A has 90 promotions in a month and 500 submitted articles vs. User B who has 30 promoted articles in a month, but just 150 submissions, that changes how you read the data. Now even if User B is getting more engagement on his submissions, their promotion rate is way higher.
I think that's actually an important piece of the data that's missing. Follow up? ;)
Posted by: Josh CatoneI agree with Josh's comment. I've often thought that these guys that I see all the time on the front page (and quite often on Diggnation) must spend all day scouring their Netvibes.com account submitting every story from every rss feed they have.
Maybe even call me crazy, but they could conceivably set up some script to auto submit every item and they simply take a quick look at the the weird-letter-security thing and enter it as they mass submit 100's of stories every day. Of course....you'd have to have absolutly no life to do that.
Never the less, incorporating the number of submissions would take this study to a new level. Very interesting, still. Good job.
Posted by: Colin Beck | May 1, 2008 8:22 PMsomehow this story got submitted to "lifestyle - educational" - is that where this belongs or was it put there to maximize the categories - as happened with the rww post last week getting fp love with 25 digg?
Mu - this is very in-depth and clearly a lot of research was put into it. I've never seen anyone correlate post-frontpage digg counts as a measure of quality.
I think it's the pre-frontpage digg counts that probably measure more towards quality.
For example, Digg made 3 tiny adjustments to their code and posted a blog entry. That blog entry made the Digg frontpage. Sites like RWW, tc, mash, gigaom, vb, cn who spent hundreds of hours working on content go without. This leads me to still believe that post-frontpage digg counts don't lead to quality.
Posted by: allen stern | May 1, 2008 8:31 PMI have never been on the front page and probably never will be. But I have a great blog.
Live From Las Vegas
Posted by: The Masked Millionaire | May 1, 2008 8:34 PMThe Masked Millionaire
Saleem,
Your posts are hardly worth the effort you put into them. Honestly, your pseudo-studies of social media sites are far from rigorous. But that's all fine and dandy if you weren't baiting your headlines to get us to read your babble.
Forrester probably has a position for you.
Posted by: eh | May 1, 2008 8:52 PMI've been a member of Digg for coming on to a year now. Never have I really been a huge user, more I was just watching, learning, having a play.
More recently however, I've started taking an interest, and by far the clearest thing to me is that they have not been as transparent as they could have been in a few instances, and I think it hurts them. For me, I now am wary of completely trusting Digg, and that I think, is a huge shame - the product is good.
Thank you for the time you spent on this Mu.
Posted by: Lid | May 1, 2008 9:42 PMI think that you did a good job with this article Mu. I really appreciate when people make arguments and have some data to back them up. There are some clear points made here but I do agree that Josh's recommendation of some missing data to complete this analysis would be # submitted items with # of promoted items. Thanks for putting in all that work. :P
Posted by: Joel | May 1, 2008 10:12 PMGreat post!
I echo strongly Joel #10. RWW is a data island in a sea of blogger buzzwords...
Posted by: Noah | May 1, 2008 10:31 PMWhat would really be intriguing would be to get some data on duplicate submissions - where stories were the same and the titles were equally compelling - but the story submitted by the top members made it to the homepage, while the story by the average member did not. Even if they submitted it first
There have been so many cases of this.
What is even more amusing is the top blogs that agree to send their posts to the top members immediately upon completion.
But sometimes, an average Digg member got to it a few minutes before it was discovered by the top members - the blog will then CHANGE THE URL of the story so that the top member can re-submit it - thereby getting them many more DIGGS.
There have even been cases of top members so angry about missing the story - they actually add a '?' at the end of the url to get it past Digg submissions
Also if you put the DIGG amount near the story - it gets many more DIGGS
Posted by: SearcH EngineS | May 1, 2008 11:17 PMShouldn't you be using median rather than average values? Means can be skewed, e.g., Zaitratsu could have just had one mega-posting and a bunch of mediocre ones, and thus the low promotion rate is justified.
Illustration here: http://billkosloskymd.typepad.com/lexicillin_qd/2007/09/mean-vs-median-.html
AFAIK, the conclusions you're drawing, while they may be true, aren't supported by the data you present.
Posted by: TheWama | May 2, 2008 1:02 AMI'm going to be reading a lot more of Diff now that I left my job.
Posted by: Paul Jensen | May 2, 2008 1:44 AMThere needs to be a hot news or breaking news category so that regardless of who submits it - power users or digg admin can promote hot stories. Digg in the past broke the news, now they report it days later. I appreciate the people that day in and day out submit great content. Excellent post MSaleem.
Posted by: Linda | May 2, 2008 2:02 AMI sit around and read papers by people like John Hagel, Yochai Benkler and Clay Shirky all day, and I have to say that the way this article is written makes little sense.
If Mu can summarize the key findings in 200 words or less in language that is understandable by someone with very limited knowledge of Digg, this discussion would be a lot more interesting.
Posted by: Ethan Bauley | May 2, 2008 8:42 AMUnlike the lottery, getting on the Digg homepage is a work of influence. You gotta have the right friends to promote your stuff (like the ones we see on the graphs above). Just like in politics, it's not the good ones that bubble up, but only the most connected ones. From that standpoint, Digg's homepage has always been of average quality. Digg's execs are right to try and change things around (since selling out seems too late now).
Msaleem, can you say you've never worked for PR agencies looking to promote their clients' work on Digg, whatever the remuneration was?
Posted by: xavierv | May 2, 2008 10:49 AMIf your analysis is complete, and accurate, why would Digg not respond to you?
Posted by: Mediaman | May 2, 2008 1:15 PMWhy would DIgg not correct, or at least justify their algorithm?
Cound it be that certain writers choose certain topis offering greater interest levels than others? And, if so, wouldn't that explain the differences?
Maybe some responders are more comfortable discussing Net Neutrality than discussing abortion.
And, horrors, maybe some writers opinions are worth more than others.
God, this bullshit is annoying. This guy spends all his time on Digg, then analyzes the shit out of it, then threatens to boycott when his itty bitty stowies won't make it to the front page.
http://valleywag.com/348346/an-open-letter-to-digg-from-top-digg-users
Posted by: Brian | May 2, 2008 3:24 PMBeen a member for a couple of years and watched as the site has gotten prettier. This led to the site slowing down on load time, but when Microsoft took over the advertising it all went to hell. Try to open several tabs - it's pathetic.
Posted by: Robert MacEwan | May 2, 2008 5:03 PMBrian, so you're saying that you didn't actually read the article, you just care who wrote it?
Or, are you saying you don't mind that the quality of digg has gone to hell?
Posted by: Skitzzo | May 3, 2008 4:51 PMI dunno, I find digg popularity almost cliquish. So in that respect, I agree with the author when he writes:
But I also think part of it is the target demographic, who are now moving on to other social media just as they are other things in life.
Posted by: Mean Dean | May 4, 2008 12:53 AM
I dont believe that Digg's passes a story to the home page without any human edit, how hard that will be to dedicate an employee to check few stories every few minutes, I dont trust them to leave if for their algorithm .. I can trust Google if they say that because it is almost impossible to review manually all the search results, so guys don't bother ourself analyzing Digg's algorithm, their home page is not algorithm controlled.
Posted by: SEO-T | May 4, 2008 4:58 PMThe number of diggs a story gets is a measure of how many diggs a story gets, not a measure of 'quality'.
I can think of cases where two stories of (IMHO) similar quality get 5 diggs and 75 diggs, largely because one submitter was a big digger and the other wasn't.
Digg's got the problem that's it's saturated the market for 'digg'-type content. I think the potential market for social news is many times bigger, but the vast majority of people don't care for apple fandom, top ten lists, and the other foibles of the 'digg crowd.' Digg is much larger than any of it's competitors now, but well-funded and sophisticated competitors as well as smaller niche-specific sites are nipping at it's heels.
Digg won't be able to take advantage of additional investment unless it can expand it's audience greatly -- so it's trying to diversify content to get a larger audience.
Posted by: Paul Houle | May 5, 2008 6:17 AMDigg experiences the same problems as society and governments. Everyone calls it a democratic system, but it isn't.
Many people believe their vote counts and it really doesn't as much as they think, which is the lie of democracy. And Digg Corporate acts like a communisitic/socialist regime where it's deciding what matters and what doesn't.
Posted by: Russell Page | May 5, 2008 8:25 AMSo the 15 word version is: Digger upset not on front page anymore, says Digg is dying now or is dead.
Yah, not buying it. Not because I'm a huge fan of Digg, but because it's easy to call out such crap. Egalitarianism sucks, sometimes, huh?
Posted by: Jeremy Pepper | May 5, 2008 9:40 AMThe number of diggs per story can be influenced by search enigne visiors aswell. Maybe the people with more diggs on their popular stories simply linked to them from other social media sites, etc., their articles on digg gained rankins on Goolgle,etc. and thus many user saw te article (and voted) who would normaly not have noticed it on the digg main page.
Posted by: Malte Landwehr | May 7, 2008 3:51 AMThis is a fascinating article and you've obviously put a lot of effort into this; however, it seems you're partially taking "correlations" and assuming connections.
Consider some factors that aren't factored into your discussion:
(1) Temporal changes in the submission patterns of each user. It's a visibility thing. Is a certain user up and coming or a veteran? Who will garner more votes?
(2) Total submissions per user -- i.e., their overall presence in the community. Top users whom have submitted more stories probably become like a gravitational body for new "friends" and votes.
(3) Patterns in the submission times of each story, for each user. Certain times of day/ days of week are better. Maybe a certain submitter lives in a part of the world where there waking hours don't mesh nicely with the best times and thus they're inconsistent with the times of day when they submit.
(4) The fact that submitters are human and thus prone to "mistakes". That is, they won't always be perfect, submitting "quality" articles every single time. In fact, who was it that said 90% of everything you do is crap? I doubt even the top submitters are always submitting quality items - it's a game of numbers. You submit a whole bunch of what you think is good/ interesting and sometimes you're just off. But Digg being a "social" site, some "friends" will vote just to be social. Non-friends will be more selective.
Posted by: jonson roth | May 12, 2008 8:17 AMcorrection: #3 should be "... their waking hours..."
So all I'm saying is that there are many, many factors to weigh, and some are probably immeasurable. That said, it IS harder to get on the HP, and a lot of very good articles get buried. The pattern is that the same sites often get buried repeatedly. (So I'm not really disagreeing with you, just looking at things a little differently.)
Posted by: jonson roth | May 12, 2008 8:24 AM