Pretty Simple: web, digital, social



When considering some of the recent online surveys that I’ve seen or been involved in setting up, I’m reminded of the saying:

Never assume. It makes an ass of u and me.

Anon

It may be hackneyed, but it does ring true for many of the observations I’ve made around surveys. Here’s my list of the 10 most common assumptions to avoid when conducting an online survey.

1. Never assume prior knowledge

An obvious one, really, but I’ve seen surveys use acronyms and jargon that I have no clue about, and then ask for comments on them. Without explaining the terms of reference, you can’t possibly expect to get useful feedback.

2. Never assume that views are polarised

Again, hopefully an obvious one but I’m often asked to create a survey where the answers are:

  • Strongly agree
  • Agree
  • Disagree
  • Strongly disagree

This is clearly missing an option – “Don’t know” or perhaps “Neither agree nor disagree”. But those who conduct surveys often don’t want neutral results, and try to omit this middle option to force people off the fence. This will, of course, skew the results and possibly even deter people from completing the survey.

3. Never assume people will read the instructions

This one has become painfully apparent recently. You can make the instructions as clear as humanly possible, but you can’t force people to actually read them. Experience has shown that people will often skip straight to the questions, disregarding any introduction or specific instruction. That means you can’t rely on people reading about the special rules and conditions that they need to know before completing the survey, and actually you should try to design the survey in such a way that doesn’t require instructions in the first place.

4. Never assume people’s response

I’ve seen a lot of surveys which appear to infer the respondent’s opinion before even asking them. People hate leading and loaded questions, and this is especially crucial to avoid if the survey is part of an important consultative process. Keep it neutral, and consider whether the survey supports both negative and positive responses with equal weight.

Running a survey with an expectation that it will back up a previously held assumption is particularly dangerous. If the results come back differently it could seriously derail your plans. Be genuine in your use of surveys and build in the capacity to actually react to the findings. A token, meaningless survey, done just to tick boxes, will often generate more anger than running no survey at all.

5. Never assume people will finish

The completion rate of a survey varies wildly according to many factors, not least of which are the length, complexity and relevance of the questions being asked. If you have questions which you especially need answers to, consider putting them towards the start, to avoid them being missed by those who lose interest half way through. Equally, if there are questions you don’t really need to ask, consider dropping them completely.

6. Never assume people will identify themselves freely

I’m often asked to make questions about personal details mandatory – so that we force people to give us their contact details, for example. Of course, this is always requested with the best intentions – it enables individual comments to be followed up and concerns addressed. But it is also completely unacceptable. In almost all instances, you should be giving the user the choice of responding anonymously. In fact, data protection laws say that you shouldn’t collect anything more than you need anyway, so you’ll need a pretty good reason to ask for these details. And if you still insist on doing so, explain to people why you need the details and what you’ll do with them, linking it to your privacy policy.

7. Never assume people will take it seriously

Is your survey idiot-proof? In other words, if someone took it upon themselves to target your survey with silly or disingenuous responses, would that ruin the results?

Any survey should come with certain caveats about its accuracy and coverage, and as I’ve blogged before, the more controversial surveys are big targets for those with devious intentions. Multiple repeat submissions and false replies can seriously skew your results, and are common if people have a strong agenda relating to the topic of the survey.

8. Never assume everyone can access it

Not everyone has access to the internet, so you may need to make alternative arrangements to gather offline responses. You might want to circulate paper copies of a survey, for example. Be aware, though, that you’ll probably need to dedicate some time to entering all of those paper copies into the electronic version, to allow the responses to be fully collated.

There’s also the issue of web accessibility – how easily the online survey can be used by individuals with disabilities or impairments. There’s some quite useful research on the accessibility of survey tools, although it’s from 2008. This is a crucial subject, as failure to make a survey accessible could mean you are discriminating against individuals.

9. Never assume that the survey is the end

Some people will happily complete a survey then forget about it. Others, however, will expect to see action. They may wish to see the results, or a summary of the main findings. They may want to know what you’re doing in response. Surveys are not a standalone process – they should be part of a wider consultative or research strategy.

10. Never assume you actually need a survey

Surveys are very easy to create nowadays, with a wide variety of free and user-friendly tools available. Hopefully the above points highlight the amount of thought that should be going into each and every survey you run. And you should also always ask yourself whether you actually need to run a survey. I’ve seen real evidence of ‘survey-fatigue’, where hitting people with multiple requests for feedback over a short period of time will seriously dent the response rate. Keep them simple, keep them relevant and keep them to a minimum.

This entry was posted in Blog and tagged , , . Bookmark the permalink.

5 Responses to Online surveys – top 10 assumptions to avoid

  1. Gary Miller says:

    Nice article James.

    You’d think that all the points you raise would be pretty obvious, but some people just never seem to learn!

    Well worth re-iterating.

    • James says:

      You’re absolutly right Gary, most of these points should be completely obvious but you’d be amazed at how many times they are forgotten or ignored in the scramble to get a survey up and running.

      I’m also currently looking at other issues such as data protection and security when running online surveys – more about those soon.

  2. Qualtrics says:

    This is a great post. Being in the survey software industry, we have tried to pass similar tips on to our users and fellow researchers.

    For those who are interested in more survey tips, you can visit our “Qualtrics University” *link removed by request*.

    Our research tool is flexible and powerful, but without these simple rules, people may miss the results they need.

    Thank you for posting quality tips on how to improve research across the board. We’ll be anxious to read more and collaborate as we post similar items, tips, and suggestions on our corporate blog at http://www.qualtrics.com/blog.

    Thanks James!

  3. Jon says:

    Will you please do me a favor and get rid of the “Qualtrics” comment link? Thank you very much for the help.

Leave a Reply

Your email address will not be published.

Browse by Category