Online Stats: Hulu Says Nielsen's Numbers are Wrong
Yesterday, we reported that Nielsen Online's April numbers showed that the number of unique streams on Hulu grew 7.9% since March, though the number of unique users dropped slightly to about 7.4 million. As the New York Times reports this morning, however, Hulu questions these numbers and argues that they grossly underestimate Hulu's real reach, which comScore, another online measurement firm, pegs at 42 million.
Nielsen's numbers are pretty close to those we have seen from other measurement firms like Compete (7 million unique visitors for April), though Quantcast, which gets its data directly from a piece of code embedded on Hulu's site, reports about 26 million. While these other companies might not agree on the exact numbers, though, most publicly available data shows that Hulu's growth has indeed slowed down in the last two months. In its own press releases, Hulu generally quotes comScore's numbers.
To gather its data, Nielsen monitors about 200,000 panel members, a technique that is clearly informed by Nielsen's method for gathering data about TV viewers. Other metrics companies use data from toolbars, ISPs, and other sources, though Quantcast also gives site owners the option to embed a code snippet on their pages that reports data directly to Quantcast (Hulu does so, for example, and so does RWW).
Stats Need Standards
The real problem here, of course, isn't even about knowing exactly how many people watched videos on Hulu last month (even though we have to admit that this discussion is quite interesting in its own right). Instead, this kerfuffle once again shows how hard it is to correctly estimate usage numbers on the web, especially in the absence of any real standards. As every blogger can easily attest, three different stats programs will give you three different numbers.
We have to take publicly available stats, no matter from which provider, with a grain of salt. In our experience, it is always worth looking at a number of different sources, and while the trends that these services show tend to be relatively trustworthy, the exact numbers are always open for debate.
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
That has got to make the ad rate negotiation a bit dicey.
Had to look up "kerfuffle"- great word.
It's a real pain to start measuring reach and impact without standards. Argh.
I have never understood why people trust the numbers from these so-called visitor statistics providers, they are obviously inaccurate.
if anyone wants to tell you how many video streams Hulu are serving, that would be Hulu, as they are the only ones to know. and if Facebook want to lie to you about how many registered users they have, they can do that too.
Comscore, Nielsen, Compete, Quantcast - I haven't found any of these to be accurate at all. Sometimes numbers are off 50-100% which just makes me think either a) the methodologies these companies are using is worthless or b) they're just pulling numbers out of thin air.
Weren't the folks at mozilla lab thinking about developing a metrics that would user data from Firefox users? If thats bundled with Fx, considering the popularity of the browser..it would be pretty reliable.
I think the Nielson sampling method borrowed from their antiquated TV ratings service should be thrown out the window per se. The web offers much better measurement tools that were not available to measure over the airwaves viewers. I think the answer is for companies to open up their log files to independent parties who will certify traffic. Essentially, Google analytics is such a service. The one thing that Nielson does with it's sample that will be difficult to do with log files is get a demographic breakdown of the views. But the day is soon coming where the database IP addresses will be matched to specific households and maybe even specific users. When IP addresses become specifically identifiable, then a third part will be able to generate demographic view info from log files that is at least as useful as any sampling done by Neilson.
These are just estimates, nothing to fret about.
Every person using statistical information needs to understand the limitations of each method. NONE are accurate; however, their value is primarily is tracking changes in THAT analytics program for any number of specific metrics.
What you most need to know is whether what you are doing is having a positive or negative effect and that your business is moving in the right direction. The trends are a guide - individual numbers not so much.