When Nielsen rolled out its latest slate of methodological changes a week ago, it seemed to address the concerns of its most important client, and television’s most important property, the NFL. After all, it was just a year ago when the league made its concerns clear in on-the-record interviews. The league’s “ongoing passion,” its league’s chief data and analytics officer Paul Ballew told reporters last year, is “that co-viewing is inaccurately measured, especially for big events.”
Within months, Nielsen announced a pilot program to test out a new ‘passive’ co-viewing measure, which per the measurement company increased viewership for the sampled events — including the Super Bowl — by an average of four percent. Now, that measure is being incorporated into the official currency.
Yet the message from the NFL is much the same as a year ago, perhaps even louder this time. Nielsen, the league believes, is focused on the wrong things.
In an interview with Sports Media Watch last week, Ballew raised more concerns about Nielsen, specifically the length of time the company has taken to address the league’s priorities like co-viewing, compared to the relative speed with which it has moved to introduce changes “that we’re not all aligned on.”
“I would say what Nielsen — from our perspective — is struggling with right now, is they’re in some cases trying to be hyper-responsive,” Ballew said, contrasting that with the structural changes that he says are taking longer to implement. “The fixes they’re putting in place, they’re rushing to market in part because they can,” while the more structural issues, “things like out of home and co-viewing, which take investment and time, are protracted.”
In a statement provided to Sports Media Watch, Nielsen said it has “worked closely with clients, sports leagues and industry regulators to constantly enhance our methodologies. This includes the introduction of Big Data + Panel, the expansion of our Out-of-Home measurement and a pilot test of co-viewing updates, which are all successful examples of collaboration that lead into the enhancements.”
The NFL was one of the collaborators on the co-viewing pilot program, and per Ballew, the league has been working with Nielsen on the issue “for the better part of a couple of years.” The addition of co-viewing to the currency is only a “partial implementation,” he said, and Nielsen is “struggling with the roadmap to go to full implementation, which will result in more accurate measurement.”
Ballew on multiple occasions described Nielsen as moving “very slowly” on co-viewing, and as “rushing” other changes to its methodology that its clients have not had sufficient time to assess or understand. Those other changes — which include modifications to weighting and universe estimates — were made at the behest of the Media Rating Council, an organization representing the networks, per a person familiar with the matter. Nevertheless, both the speed of the changes and reliability of the resulting data are concerns for the league.
“The limited data we have raises a lot of questions,” Ballew said. “The results we’ve seen in some cases, the limited data that they send us, have negative impacts on sports,” with those offset by the impact of co-viewing.
If it all comes out in the wash, and the topline number is positive, isn’t that enough? No, says Ballew. “All of us want higher numbers, but we at the league pride ourselves on really stressing accuracy and integrity in understanding the results because we use it for all of our operational decision making and for so much of what we do.”
“The data being logical and having integrity is a very important part to the equation for us and how we run our operation,” Ballew said. “So yes, if the numbers are positive versus negative, you take it. Our concern, though, is much broader than that, and Nielsen’s actually had that argument with us. Say, well, this could all be a wash. Our response is that’s fine, and I’m glad it could be a wash. But we’re still trying to make sure that we’re getting to the most accurate result out of all of this because it’s so interwoven to what we do each and every day.”
The biggest critique of Nielsen’s “Big Data + Panel” methodology, which rolled out last fall, concerns the accuracy of the demographic picture. Even if the topline viewership number is in line with or surpasses expectations, the underlying demographic data can show unrealistic swings, according to critics. The Video Advertising Bureau, an industry group representing the networks, released a report last year arguing that “Big Data + Panel is unstable, unpredictable and decimating demographics,” as noted in a previous piece. (Nielsen later described the VAB analysis as “seriously flawed and manipulated.”)
In a recent conversation with Sports Media Watch, Nielsen SVP/product strategy and thought leadership Brian Fuhrer said the company is addressing the stability of demographic data with its recent methodological changes.
“As we were analyzing the comparison of panel of Big Data and looking at specific demographic groups, there were certain parts of the weighting from a person’s age group that we feel like could have been improved, and our clients were very generous in providing us that input as well,” Fuhrer said earlier this month. “So we went back and looked at the model and made some adjustments to more accurately reflect the different demographic groups across Big Data. I don’t think there’s going to be any big changes to the totals or anything like that, but I think overall, it’s going to be more consistent and stable across the demographic groups.”
While much of the conversation about television ratings focuses on whether the numbers are up from the prior year or ahead of the competition, arguably the greatest value that Nielsen provides — and that which separates the company from its competition — is the demographic data that drills down into specific groups, from age and gender to income. That is data networks, leagues and especially advertisers rely upon to make decisions about where to invest their time, money and resources. For the NFL, which strategizes its schedule perhaps more than any other league, that data helps determine where and when games are played.
“We want to make sure that we’re operationalizing the games, where we’re putting them, what networks, what audiences we’re reaching,” Ballew said. “Because the NFL’s commitment is to reach the broadest audience possible in the United States as well as in the world, and when we don’t have that information, it creates blind spots for us in terms of us continuing to optimize our game and our game schedules.”
Would the league follow in the footsteps of NASCAR, which earlier this year stopped reporting “Big Data + Panel” and reverted to publicizing the prior panel-only standard? “No, we would not. At the end of the day, Nielsen calls the balls and strikes,” Ballew said. “What NASCAR did, I’d leave to NASCAR. We, like most, accept what the currency is in the marketplace, and we try to make it better.”
“But along with accepting it as the currency,” he added, “we believe the numbers continue to underreport live sports and especially the NFL,” particularly in “key moments” like the Super Bowl, earlier rounds of playoffs and Thanksgiving. “So, we accept it as a currency, but we will continue to articulate the fact that we believe we’re being undercounted, and it’s in millions of viewers.”
For how long will the NFL be so accepting? Ballew said the league is “continuing to look at other alternatives to Nielsen,” citing the importance of improving measurement in a changing media environment. He described that effort as “accelerating,” owing to the lack of progress in some areas the league feels need attention, and the rush to change other aspects on which the league is not aligned. “It’s not something we feel reflects positively on Nielsen as the currency, so we want to make sure we have options.”
Ballew said the NFL is looking at “a host of companies and a host of different ways to help them improve their measurement.” Though he did not specify any platforms, he previously mentioned VideoAmp on a call with reporters a year ago, describing the company as “intriguing.”
The league is also continuing to work with Nielsen, per Ballew, but the progress in his view has been limited. “We’ve invested so much time helping Nielsen over the last few years, and we’re sincerely committed to just improving the accuracy of the measurement. And it’s an interesting situation to be in going into the season where we had a lot of hope, given the progress we were making, and it’s been a bit of a two steps forward, two steps back journey — which is what’s a little unsettling to us, to be honest.”
Nielsen, as one might expect, sees the situation a bit differently. “I can tell you that the enhancements that we’re putting in from a data user standpoint this fall are very solid,” Fuhrer told SMW earlier this month. “I think all of them are really substantial improvements to the data. The challenge is putting them all in at the same time.”
Ultimately, Fuhrer added, “I think we’re coming to a very logical place that I hope the industry can really coalesce around.”
Ballew said he is “empathetic” that the job of Nielsen is “fairly complex.” The company, he added, is wrestling with the “fundamental challenge” of relying on its panel as a foundation in an era where “the world has moved away from surveys and panels,” likening it to changing a tire while the car is still moving.
The solution in his view is to focus on the structural changes the league has prioritized, including co-viewing, more first-party data integration (which he described as “unbelievably protracted”) and expanding the “Big Data” collection beyond its current 1/3 of U.S. homes. Those issues, plus quality assurance processes — Nielsen had to revise and re-issue its Super Bowl viewership figure for the second time in four years this year — are the “areas we’ve been working with them on.”
“If you’re in the data business, which is what they are, you need to get the fundamentals right, including validation and quality assurance,” Ballew said. “And then on top of that, you have to continuously innovate. The world we live in is a hyper digital world, and the world we live in has moved far beyond just being panel and survey.”
