Market Research – Pity the Respondent

02 June 2014

We recently had a Bank Holiday. Traditionally this is when the nation’s men pile off en masse to the DIY store, buy a load of gadgets they’ll use once at most, rush home, put the gadgets in a drawer where they’ll stay for some time, and settle down to watch sport on the TV. But this year was different – at least for me. For this was the year that Bank Holiday Monday was Market Research Day.

It’s been a while since the Cog Blog commented on the world of market research. Last time around, my slightly critical post (I suggested that maybe the research business was somewhat out of touch with its customers and their current needs) generated some heat and (for me) a huge audience. The lure of the audience number beats my trepidation in risking upsetting the researchers, so here we go again.

Over the holiday weekend and the following few days I was asked to fill out an online questionnaire by British Airways; then another one from The Observer newspaper. I always try to complete genuine online studies (oddly I’m never asked, as used to be the case, whether I have any professional interest in market research). Increasingly the questions leave me looking for the most appropriate way to express my thoughts or views within an inflexible pre-coded structure with minimal instructions.

An example – BA wanted to know what I thought of their arrival procedures. Now, I no longer have the questionnaire and it could be that I’m doing the BA researchers a dis-service here but what on earth are they talking about? It can’t be anything to do with the bit after you get off the plane, as that’s nothing to do with BA. How to score my feelings about the BA arrival procedure?

So I marked everything neutral and wrote in the comments box ‘the plane arrived and I got off. It seemed to go OK’.

Then there was The Observer asking me to score them on various ethical dimensions by category of interest. So – do I think their articles on beauty are ethically appropriate? What, all articles on beauty? Are we to assume that all are ethically identical?

The key rule is rubbish in, rubbish out. I really had no idea what The Observer, or BA were getting at with several of their questions. I finished both surveys as quickly as I could and went back to my DIY (as I call the TV).

Also over the holiday weekend a friend of mine’s husband (I’m not going to give a name for obvious reasons) was completing the TGI questionnaire. This takes days, weeks even. My friend, who was involved in the marketing business until reasonably recently and therefore knows a bit about the survey was good enough to provide me with a running commentary on this task.

Phrases like ‘losing the will to live’; ‘these questions are unanswerable’; ‘where’s the n/a box?’ and ‘you could make a comedy sketch out of these lifestyle questions’ all appeared. I would add: ‘does anyone working for TGI actually ever complete the TGI questionnaire?’

A little while ago I was asked to help a client as they committed to an integrated media study called MCA. I asked the client and agency team to complete the questionnaire, and their answers dumped before ‘real’ people went through the process. Personally I found completing the survey an awful experience – repetitive, boring, hard to complete, and it took an age.

Does anyone actually ever test these questionnaires before sending them out to punters? I know this used to happen but I bet it doesn’t happen so much now. Does anyone ever ask respondents to report back on their experiences with the questionnaire, having completed it?

Back in the dim and distant past I once had the opportunity to visit some BARB (in those days JICTAR) panel members. They were about to become ex-members due to boundary changes, although they didn’t yet know that, and their data was already being excluded. Their feedback was really interesting, not least the seriousness with which they treated the task of recording their TV viewing. They felt they were genuinely contributing to what appeared on TV – and to an extent they were of course correct.

Someone pays for the research. Someone analyses the results. Someone takes what respondents say seriously. As a respondent I try to take each task seriously but it seems to me that increasingly the researchers aren’t helping.

My point is that in each of the BA, Observer and TGI examples the respondent experience was dire. I’m sure a researcher will point out that there is an n/a box on the TGI (I don’t know if there is or there isn’t), someone is bound to tell me that the MCA study brings great benefits to users and I’ve no doubt that BA or The Observer will point out that their studies were easy to complete and that therefore by definition I must be an idiot.

Market researchers do need to think about the poor respondent. Do test run studies. Do collect some feedback. Don’t just repeat the same questions over and over again whilst chanting the researcher’s sacred mantra of compatible data. After all, rubbish compatible data is still rubbish.

It’s reasonable surely to ask whether anyone can really answer the questions accurately and truthfully. Test it on yourself, and on your clients.

A better respondent experience must mean better market research. And on a scale of 1 – 5 I would definitely agree that that is a good thing.