Edward Appleton: Can Researchers Trust Online Access Panel Data?


Dies ist ein Beitrag von Edward Appleton, Research & Reflect.

I recently returned from a 2 day MRMW conference about mobile in market research in London. It was in one key aspect an eye-opener.

As a backdrop: the overriding focus was on how technology providers have ingenious plans to further „transform“ Market Research – recruiting respondents differently through gaming, using beacons for „in-the-moment“ insights, accessing emerging markets through low-tech mobile devices, reducing survey length….Some were more convincing than others, lets say. Common denominator? Scaleability…..

Very few provided powerful examples – case studies – of how mobile had actually helped surface insights that otherwise wouldn’t have been possible.
The session that had the strongest impact was one lead by Survey Sampling International’s Global Knowledge Director, Pete Cape. He had invited 12 „real respondents“ into the Conference room, and asked them to give their honest views on market Research.
What they said was both fascinating and horrifying:
– money was often the first-mentioned motivator for MR participation
– many openly confessed to multiple Panel participation
– not telling the truth was openly admitted
– taking upwards of 30 Surveys per month (one gent had done 80 in the last 4 weeks) was commonplace.

Many were also extremely critical of Survey design – „most of your mobile surveys are total c**p“ was one succinct statement, grids were torn into, many surveys simply not working technically…..

It was a sorry state of affairs all round – Market Research being gamed by savvy pros interested in the money, disdainful of the people responsible . Curiously, the audience loved it. Much laughter. A distinct lack of embarrassment – very odd.
I was aghast, and tried to seek out Mr. Cape at the SSI stand to understand his motivation – why on earth would he do such an apparently self-destructive thing? Sadly, he was nowhere to be seen, so I am guessing as to his strategic intent.
Despite not having hooked up with Mr. Cape – yet, and perhaps he’ll comment on this blog if he reads it – here’s my take:

Online Access Panels – Change Afoot?

I can only assume that SSI deliberately wish to disrupt the business model of online Access Panels.
Public confessions of misdeeds can be therapeutic, but in a business context they are likely (hopefully?) more purposeful. SSI effectively let the world see that response quality in online Access Panels is very poor indeed. Why should anyone wish to buy that sort of low quality – regardless of price?
If you are an end-client or a Major MR agency, why on earth would you wish to continue to simply purchase „professional“ responses? Or be associated with that in any way?

An Open Secret isn’t any less Shocking

Many people in the audience were less shocked than myself – shoulder-shrugging seemed the most common reaction – that’s the way it is, seemed to be the reaction…. „Good on Pete for outing it“ „You look shocked, Edward – are you going to write a blog about it?“ „It’s true what the respondents said, I worked for years for an Access Panel…..“
Does nobody care very much? Is there such a disregard for quality in large areas of our quant. sampling industry that the concept of disruption seems to leave people unmoved? Or are the financials so ugly that a shake-out is possibly even welcomed?
We often talk and write feverishly about the future of MR – but what about the present?
At a guess I would imagine that at least 50% of all quant. survey work uses access panels – what about all the major MR companies such as TNS, Ipsos, GfK with reputations to manage – is this all so new?
Take it to the client level: what about all the multi-million pound decision making taken on the basis of this….“c**p“? Do they know about this slight quality issue?
UK retailer Gerald Ratner’s remarks confessing one of his jewellry products was „total c**p“ had catastrophic consequences for his eponymous jewellry chain, as some in the UK might remember.

„It’s True…but Keep it Amongst Yourselves“….Beg Pardon??

I checked out Social Media – Twitter first and foremost – for mentions of the disruptive impact of the SSI Piece, using the Conference hashtag #MRMW. The result: very few mentions indeed.
Twitter of course isn’t representative of a MR universe; perhaps the topic is too explosive – or perhaps people simple would prefer to pretend the session hadn’t happened. Just shut up and „move on“. A conspiracy of silence?

So what’s my take-out?

Well, recruitment quality is and has been an issue in many areas of the Research Industry for decades.
But if what SSI’s „real respondent“ session suggested is in any way representative of the online Access Panel universe, then it has broad implications – „Quality of Response“ has been „outed“ as very very low. Online access panels aren’t robust.
Worse: many of the more sophisticated parts of the MR toolkit relating to attitudinal measures – derived importance, conjoint, predictive analytics, mixed-modal implicit/explicit measurement, Bayesian statistics….- are pretty irrelevant if the fundamentals are effectively rotten. As for mobile? It hardly matters if it’s „in the moment“ if that’s a fake moment.
„The King is dead“ is normally accompanied by the phrase „Long live the King“ – so I will watch the SSI space and see if the disruptive session was actually deliberate, launched with strategic intent, to see „what’s next.

If not, or if no further explanation is forthcoming, my blog floats off into the ether unnoticed, then I would suggest that the legitimacy of such online Access Panels is seriously questioned – and as such should come with a clear health warning to all future users.
I do hope I was not the only person to be shocked by the session – naivety to me is preferable to cynicism.

Curious, as ever, as to others‘ thoughts.


Über den Autor

2 Kommentare

  1. Hi Edward,
    thanks for sharing! It is – in the end – the question about transparency for the end-client who wants to derive the right conclusions and draw good business decisions. I have been on the companies side like you – and often heard those rumours that were spread from agency side. I felt they were close to criminal behaviour when they talked e.g. of questionnaire duplication – but on the companies side we never knew the right questions to ask or had the right tools to get a final check on the true dimension of the issue. Do we talk about few black sheep? Who is in control of the situation, how big is the issue? Who is able to give the end client a „quality seal“ – which would probably also impact the dimension of prices paid…good prices for good quality?
    We should think about how to re-establishing the credit of market research services – both sides agencies and buyers of market research services.
    Worth spending more thoughts on this!

  2. Hi Edward,


    „Just shut up and „move on“. A conspiracy of silence?“

    I believe a reason for no interest in revealing this and almost no reaction is that it is lowering the quality of work done in MR and damaging the reputation of everyone employed in MR. Until there isn’t some new disruptive / dogmatic new way of ensuring data quality in access panels to follow, there will not be much that will step forward without a solution.

    How to develop a solution for this problem is a task for our whole industry.


    P.S.: shared this on my social media accounts 🙂