Mobile News Mix

How Failing Online Panels Put Your Data in the Dumpster

Posted by admin on Jun 13, 2017 9:41:24 AM


Blog Pic Panel Quality Crisis


If your job is consumer insights, your every working day revolves around avoiding the fatal trap of data analysis: “Garbage in, garbage out.” Right?


Well, not always. Too many researchers for too many years have been accepting garbage from online panels because it’s the familiar and expedient thing to do. They’re willing to partner with online sample providers who resell and recycle the dwindling numbers of consumers who are still willing to take surveys on laptops and desktops, including legions of “professional” survey takers who play havoc with data quality. Not even a data-analysis genius can guide executives to good business decisions if what’s being analyzed is poorly sourced. But now we’ve reached a hopeful moment, a chance for researchers and brands’ insights departments to wake up, smell the trash, and bury it for good.


The recently-published GreenBook Research Industry Trends Report (GRIT) for the first half of 2017 reinforces a call to action that GreenBook’s editors have been making for more than a year: with online panels in crisis, it’s time to stop shrugging and start improving. But inertia is a powerful force. Similar alarms have been sounding for years, mainly unheeded.


Ron Sellers of Grey Matter Research & Consulting loudly blew the whistle back in 2009, in a hard-hitting report entitled “Dirty Little Secrets of Online Panels (And How the One You Select Can Make or Break Your Study).” He recruited his own team of “mystery shoppers” who each joined a dozen leading online research panels and took surveys for a month, recording their experiences in detail. Sellers and his helpers gave researchers a sophisticated, systematic look at how panelists were being recycled through multiple surveys – without regard to the fact that this kind of repetition and churn sacrifices quality to quantity, taints the data, and burns out the respondents.


Some of the panel providers Sellers included got passing grades, but others were caught in the act of supplying garbage-in sample for the sake of filling survey quotas. Among the unhealthy practices the study documented were packaging panelists from multiple sources without identifying them for a client, and biasing and burning out panelists by filtering them through multiple surveys. Consider these “Dirty Little Secrets” from 2009, and ask yourself whether they still ring true eight years later:

  • “Not only are panel members inundated with multiple invitations [nearly two a day, on average, for two well-known panel providers]...but there appeared to be no limit to the number of invitations a panel member could receive or the number of studies in which they could participate.”
  • “Over half of the surveys our shoppers attempted for Panel 3 were actually for Panel 12. Our shoppers even had the opportunity to complete the same questionnaire for this panel company multiple times through multiple different panel providers.”
  • “End clients are getting what is likely to be poor quality research and limited insight from overused respondents. The critical question is: are you one of them?”
  • “Apparently, some researchers either remain blithely unaware of what they’re getting, or they don’t care about data quality.”

Sellers repeated the exercise in 2012 and found that little had changed. It's even more telling that 45% of research clients surveyed for the just-published edition of the GRIT report said that panel quality continues to decline, versus just 20% who thought it's improving. How long will the insights industry tolerate this race to the bottom?


There's one crucially important opportunity that makes today much brighter than 2009 and 2012, if you're willing to seize it. Its the rise of mobile research and its ability to draw in a large, engaged and validated mobile panel. State-of-the-art mobile surveys take place inside a research app, making it possible to recruit a large, diversified and responsive panel because smartphone users want smoothly functioning mobile apps, not slow, frustrating sessions waiting for a mobile browser to connect with a website. There’s no need to recycle or outsource panelists when they flock to download a mobile survey app. Innovative mobile technology breeds strength in numbers, strength in engagement, and strength in data. Quality in, quality out.


Want to be part of the panel crisis’ solution? Sunlight is the best disinfectant, so let’s start a panel-quality cleanup right now by making transparency a priority and bringing specific issues to light. If you have a story to share, we’d love to hear about it. Just get in touch at

Topics: MFour Blog

Post a comment

Subscribe to Email Updates

Recent Posts

Posts by Topic

see all