Don’t Let Online Panels and Data Leave You Shortchanged  

Posted by admin on Jul 19, 2017 10:00:34 AM


Blog Data Quality 900 x 300


If you’re not sure why you should pay close attention to consumer panel fraud, just think back on “The Big Short.” Nominated for a 2016 Oscar for Best Picture, it told the story of the subprime mortgage meltdown that triggered the Great Recession of 2008-2009.


Now it’s market research that’s facing a meltdown, if researchers and clients don’t make it a top priority to know exactly where their data comes from, and how vulnerable it will be if it’s sourced online. It’s this issue that brings to mind “The Big Short,” because the movie is a sobering study of what can happen if an industry gets too complacent or too distracted to be vigilant about data sourcing and data quality.


The film tells a complex story that boils down to a pretty basic business mistake: a failure to obtain reliable consumer data before making important decisions. “The Big Short” focuses on investment bankers who bought thousands of individual home mortgages, bundled them into mortgage-backed securities, and resold them to investors. In the rush to exploit an opportunity created by booming home prices -- and the belief that they would never come down – many sharp minds on Wall Street failed to pay attention to the basics. As the movie shows, they failed to obtain or verify actual on-the-ground facts concerning the financial qualifications of the real home-buyers who were taking out the actual mortgages that served as the foundation for mortgage-backed securities. 


Suspecting the worst, a hedge fund executive played by Steve Carell heads to Florida to do the ground-level consumer research the sellers and buyers of mortgage-backed securities had fatally omitted. He visits actual homeowners in actual neighborhoods to learn how much they owe on their mortgages, and how much they earn. He’s appalled to find that they have no chance of paying back the loans, making it inevitable that they will default, and that the investment pyramid built on their limited resources will crumble and fall.


Today’s consumer researchers and their clients now have to decide whether to risk a Big Shortchange in data quality and reliability. If they opt for online surveys, as most of them have for a generation or more, they’re likely to be buying consumer sample that’s in some ways akin to those mortgage-backed securities. Instead of known respondents who’ve joined a single, carefully-managed panel, buyers of online sample typically get data from a mix-and-match set of respondents who come from a variety of sources. They get verbal assurances that the data is representative, but no transparency about how it was sourced. This leaves the process open to all sorts of errors and abuses that undermine executives' ability to base important business decisions on accurate, reliable data.


One risk of relying on an online panel is duplicate respondents. Because many online panel members belong to multiple panels, there’s a real danger that they’ll receive multiple invitations to take the same survey. Two completes from one panelist is in nobody’s interest. Online research also is vulnerable to fraud by survey-taking bots. Experts say that bots are getting better all the time at mimicking real respondents and evading detection.


The alternative to online surveys is in-app mobile studies that bring two core benefits: They harness consumers’ love of their smartphones to insure a diverse, representative panel, and they exploit smartphones’ unique capabilities to validate each response and ensure it’s not coming from a bot or from duplicate survey-takers. Each phone has its own unique ID code and can be located geographically for further assurance against duplicate responses. Meanwhile, fraud-bots that stalk the online realm can’t break into the protected, in-app space.


If you want to invest in the most reliable data to inform your client's or company's analysis and decisions, it’s important to understand how legacy online technology and methodology stack up against new-generation in-app mobile solutions. To get in on the conversation, contact us at

Topics: MFour Blog

Subscribe to Email Updates

Recent Posts