Survey biases
Let’s look in detail at survey biases in market research. Survey biases are systematic errors introduced into sampling or testing by selecting or encouraging one outcome or answer over others. While B2C research is straightforward when addressing people on known topics, B2B research is more complex. The real difficulty is to measure perceptions on terms that are not always well understood.
Survey Biases in Market Research
Since online surveys (or CAWI for Computer-Aided Web Interviewing) have democratised access to marketing research, almost anyone can go into the field and design their own survey.
When designing a survey, however, it is important to avoid a number of errors known as ‘survey bias’, which have existed since long before the Internet.
Main Survey Biases
One of the main survey biases, if not the main one, is to ask profile questions (age, gender, location…) at the beginning of the survey when they are really meant to be asked at the end. These questions can in fact dissuade respondents from answering as they can be considered to be too intrusive when posed at the beginning of the survey. The second bias is to administer a survey without having first run it through a small group of respondents.
It is then necessary to avoid double-barreled questions, that is, questions in which two questions are asked. For example, ‘Do you think Martinique is a nice and cheap destination?’ A person may think that Martinique is a nice destination but not cheap, or vice versa…
Another Bias
Another bias is often found in questions that already include the answer or at least the opinion of the person who wrote the question (or someone else): ‘Would you buy this fridge despite its design flaws?’ Hard to answer objectively…
One should also be careful of halo effects: ‘Do you also think, like the President of the Republic, that taxes are too low?’ Referring to a known person undermines the neutrality of the question.
Designers of surveys should avoid the use of jargon: ‘Would you use this DBMS in your professional life?” Terms that are too generic are recommended to be banned. For example ‘Would you use dematerialisation to process this task?’ It is necessary to specify what is meant by dematerialisation: software, a specific procedure…
When using a list of responses, the list should be as exhaustive as possible or include an ‘other: please specify’ or an ‘I don’t know’ choice. Example:
What is your favourite dessert?
- Strawberry Trifle
- Apple pie
- Chocolate éclair
The person who prefers Paris-Brest will be frustrated not to see their favourite dessert mentioned and not to be able to choose the answer ‘Others’.
When talking about distances or times, it is important to be as precise as possible in quantifying them. Example: Less than 5 km / Between 5 km and 10 km / Between 10 km and 50 km / More than 50 km.
Instead of: A short distance / A medium distance / A long distance / A very long distance.
Tips
Also, be careful not to put too many open-ended questions in a survey.
Closed questions are favoured due to being the easiest to process and quantify.
Open questions, on the other hand, require a more qualitative treatment. Although they are indeed necessary, they are not as frequent.
In addition, it is important to assume that some respondents might not know how to answer a question. Therefore, it is essential to create an ‘I don’t know’ or ‘Prefer not to answer’ option for those people. Otherwise, you may end up with multiple responses chosen by default.
Double negatives should be avoided as they mislead the respondent. Example: Do you think that cars and aeroplanes would not be safe as means of transport?
Limitations of Studies Due to Survey Bias
Online surveys allow for a big group of people to be surveyed on a given topic at a low cost. However, as with all self-administered surveys, there are some downsides which must be taken into account:
- Interviewees may misunderstand some questions
- Responses to open-ended questions may be poorly expressed. This may pose processing problems later.
- The same person may respond multiple times if unsupervised.
- If the pollster has to rent a panel to reach its target, respondents may not be motivated by the questionnaire. If there is payment for their response, respond too quickly.
- Some categories of people or age groups are difficult to reach via the Internet.
Trends and Innovations
To make a questionnaire attractive, in addition to the biases described in this definition, one should avoid creating a questionnaire that is too long.
How many questions should you offer to your future respondents? It’s hard to answer precisely, but if a respondent spends more than five minutes on your questionnaire, that’s already very good!
If you really need to ask a lot of questions, it is better to split the survey in two and organises two separate campaigns.
Of course, the more relevant your survey is to your target group (i.e. the more the respondents feel concerned by the questions asked), the more valuable answers you will collect.
Finally, be completely transparent with your respondents about the objectives of your questionnaire. For example, you should tell them very explicitly that you are carrying out a market study on estate agents. That you are interviewing them as experts in the sector. Reassure that their answers will remain confidential. Send them a summary of the study once you have written your report. Your transparency will reassure and motivate them. You will create a real relationship of trust with them.
Tools and Methods
Surveys can be semi-structured or structured. They can be administered either face-to-face, by telephone, by post or online (CAWI: Computer Aided Web Interviewing).
Directional questionnaires usually consist of about 20 questions on a specific topic. The questions are either closed or open-ended, ranging from the most general to the most specific.
Links and resources on survey bias
