When considering some of the recent online surveys that I’ve seen or been involved in setting up, I’m reminded of the saying:
Never assume. It makes an ass of u and me.
It may be hackneyed, but it does ring true for many of the observations I’ve made around surveys. Here’s my list of the 10 most common assumptions to avoid when conducting an online survey.
1. Never assume prior knowledge
An obvious one, really, but I’ve seen surveys use acronyms and jargon that I have no clue about, and then ask for comments on them. Without explaining the terms of reference, you can’t possibly expect to get useful feedback.
2. Never assume that views are polarised
Again, hopefully an obvious one but I’m often asked to create a survey where the answers are:
- Strongly agree
- Strongly disagree
This is clearly missing an option – “Don’t know” or perhaps “Neither agree nor disagree”. But those who conduct surveys often don’t want neutral results, and try to omit this middle option to force people off the fence. This will, of course, skew the results and possibly even deter people from completing the survey.
3. Never assume people will read the instructions
This one has become painfully apparent recently. You can make the instructions as clear as humanly possible, but you can’t force people to actually read them. Experience has shown that people will often skip straight to the questions, disregarding any introduction or specific instruction. That means you can’t rely on people reading about the special rules and conditions that they need to know before completing the survey, and actually you should try to design the survey in such a way that doesn’t require instructions in the first place.
4. Never assume people’s response
I’ve seen a lot of surveys which appear to infer the respondent’s opinion before even asking them. People hate leading and loaded questions, and this is especially crucial to avoid if the survey is part of an important consultative process. Keep it neutral, and consider whether the survey supports both negative and positive responses with equal weight.
Running a survey with an expectation that it will back up a previously held assumption is particularly dangerous. If the results come back differently it could seriously derail your plans. Be genuine in your use of surveys and build in the capacity to actually react to the findings. A token, meaningless survey, done just to tick boxes, will often generate more anger than running no survey at all.
5. Never assume people will finish
The completion rate of a survey varies wildly according to many factors, not least of which are the length, complexity and relevance of the questions being asked. If you have questions which you especially need answers to, consider putting them towards the start, to avoid them being missed by those who lose interest half way through. Equally, if there are questions you don’t really need to ask, consider dropping them completely.
6. Never assume people will identify themselves freely
7. Never assume people will take it seriously
Is your survey idiot-proof? In other words, if someone took it upon themselves to target your survey with silly or disingenuous responses, would that ruin the results?
Any survey should come with certain caveats about its accuracy and coverage, and as I’ve blogged before, the more controversial surveys are big targets for those with devious intentions. Multiple repeat submissions and false replies can seriously skew your results, and are common if people have a strong agenda relating to the topic of the survey.
8. Never assume everyone can access it
Not everyone has access to the internet, so you may need to make alternative arrangements to gather offline responses. You might want to circulate paper copies of a survey, for example. Be aware, though, that you’ll probably need to dedicate some time to entering all of those paper copies into the electronic version, to allow the responses to be fully collated.
There’s also the issue of web accessibility – how easily the online survey can be used by individuals with disabilities or impairments. There’s some quite useful research on the accessibility of survey tools, although it’s from 2008. This is a crucial subject, as failure to make a survey accessible could mean you are discriminating against individuals.
9. Never assume that the survey is the end
Some people will happily complete a survey then forget about it. Others, however, will expect to see action. They may wish to see the results, or a summary of the main findings. They may want to know what you’re doing in response. Surveys are not a standalone process – they should be part of a wider consultative or research strategy.
10. Never assume you actually need a survey
Surveys are very easy to create nowadays, with a wide variety of free and user-friendly tools available. Hopefully the above points highlight the amount of thought that should be going into each and every survey you run. And you should also always ask yourself whether you actually need to run a survey. I’ve seen real evidence of ‘survey-fatigue’, where hitting people with multiple requests for feedback over a short period of time will seriously dent the response rate. Keep them simple, keep them relevant and keep them to a minimum.