Impact of Question Design on Research
The addition of explanations and including a multi-select category in/before a question are two case studies that illustrate the impact of question design on research.
CASE STUDY: ADDITION OF EXPLANATIONS
- In this case, survey participants were asked the following question: "please try to recall a recent instance in which you found important information online, information that led to a significant action or decision. Please describe that incident in enough detail so that we can visualize the situation."
- The question also defined a significant action or decision as a meaningful change in one's plans, thinking, or actions.
- The results showed responses such as "I looked at the weather app on my phone before I left for work in the morning. It said the temperature was colder than I expected it to be. So, I put on a warmer coat and a hat."
- Another respondent in search of a job provided details as to how she changed her job application strategy to focus on specific companies due to online information.
- The data only revealed results regarding specific changes, even though a change does not necessarily depict a significant action or decision.
- The reason for this result was the addition of explanations to what significant in the question entailed, which eventually led to bias in the respondents' responses.
CASE STUDY: INCLUDING A MULTI-SELECT CATEGORY
- In this study, the question asked respondents to detail significant activities that they might have carried out online within the past month. However, the question also included a multi-select category before the main question.
- The multi-select category asked the respondents to select options for their online activities within the past month, such as "bought something, watched a TV show or movie, planned a vacation, sent an email, posted on social media (for example, Facebook or Instagram), and researched a topic."
- The answers to the question regarding significant activities carried out online by the respondents were very similar to the options provided in the multi-select category, with most of the participants reporting details of research they did online.
- Some responses included "researching information about tax base transfer in California" and searching for weight-loss surgery information online.
- The multi-select category and, more specifically, the option for researching a topic online streamlined the nature of respondents' answers, as it primed their responses for significant activities to represent researching something serious such as health and sector-related information.
- However, significant activities online do not ordinarily refer to only research. As such, the multi-select category had an obvious impact on the responses from the participants.