top of page

Bias Beware

In today's world, survey research has become more and more democratized. Free on-line tools like SurveyMonkey are readily available to conduct surveys to identify transit rider pain points and track progress towards remedying them.


But be aware of bias. While it is helpful to have tools that don't require advanced knowledge of research methodology, programming languages or statistics, be aware that these tools may not produce trustworthy information. This is especially important for "tracking" surveys that measure changes over time in rider satisfaction with service quality, safety, cleanliness, ease-of-use for apps and websites, etc. These results are often used to evaluate the performance of staff, so take extreme care to get it right!

person sitting at desk reading survey results

In previous eras, surveys were almost universally conducted by market research firms with deep knowledge of survey methodology and statistics. They used random digit dialing, random samples of addresses, and other techniques to avoid bias. Market research firms still keep busy today, but free on-line tools provide a Do It Yourself (DIY) alternative for people on a low budget. Unfortunately, Do It Yourselfers don't always have the market research expertise needed to produce information that is accurate enough to inform consequential decisions.


DIY surveys, for example, may not use random sampling techniques to avert bias. For surveys on-board a transit vehicle, for example, surveyors might select people they feel most comfortable interviewing even without being aware they are doing so. They might oversample or undersample particular ages, genders, or races. This is sometimes called unconscious bias. Or surveyors might neglect to survey certain groups of riders like customers who are blind or deaf because it requires extra effort. All of these biases can be minimized through careful training and supervision.


Biases can also occur in on-line surveys. A DIY survey might get posted on the internet as an open survey for anyone to fill out, but this can create a sample that is not representative of the population of interest or excludes certain groups. Open on-line surveys may produce huge sample sizes, especially if supported by advertising on Facebook or other channels. But large sample sizes should not be mistaken for accuracy. If the sample is not representative and inclusive, it doesn't matter how big it is.


Another potential source of bias is the questionnaire itself. Questions have to be carefully worded to avoid bias. For example, in market research you should never ask "leading" questions like "Do you like the modern new bus you are riding today?" Questions must be worded neutrally to prevent bias, e.g. "Please rate the bus you are riding today on a scale of 1=Poor to 5=Excellent."


Tips: For transit DIY research, take a course in survey research to learn the basics. Even a beginner course will provide the essentials to help you create a sampling plan and questionnaire that avert bias. For transit agencies that can afford to engage a market research firm, write a scope of work that requires the firm to submit a random sampling plan for your approval, and require them to disclose methods they will use to avert bias. Also be sure to attend surveyor training sessions to emphasize the importance of producing accurate results.


Even experienced firms can get it wrong. Think about polls that incorrectly predict election results for example. Nevertheless, experienced firms are your best bet to obtain accurate research, especially if transit staff carefully monitor their work at every stage.


Testimony from people who come out to public meetings or focus groups or tweet to their followers can also be biased unless the speakers are selected at random from the general population. The same is true for "outreach" to get feedback on projects or to comply with regulations. If the participants are not selected randomly, the results will not be accurate. In fact, these outreach efforts should not be called surveys. It is more honest and transparent to call them "outreach questionnaires."


If all of these biases aren't bad enough, research consumers often introduce their own preconceptions and filters when they see or hear research results. Some listen selectively and filter out any findings that conflict with their ideas or values. Others may disregard the research entirely and substitute their own gut feelings.


Tip: Talk openly with your audiences about all forms of research bias (including selective listening), and the accuracy level of the information you present. It's best to be transparent. If results are based on non-random samples, publish a Caveat that starts with "Findings may not necessarily reflect the population at large..." People will appreciate your candor. Conversely, if you use a robust, valid methodology, tell your audience about it so they know they can place their trust in the information you provide.

50 views

Related Posts

See All

Comentarios


bottom of page