Science of Surveys Step 2: Conduct question brainstorming and pre-testing.
In this step, analyze each objective and write as many questions as you
can think of related to that topic. Don't worry about redundant or irrelevant
questions at this stage, just get down as many ideas as possible. When
writing questions, here are a few common mistakes to avoid*:
1. Too few/ Too many questions 2. Too many "open-ended" questions 3. Too much attitude, not enough behaviors 4. Poorly worded questions
Mistake #1: Too few/ Too many questions
One of the toughest problems for survey developers is finding the optimal
number of questions for the survey sample. If the survey contains too
few questions about a topic, it will provide an incomplete or inaccurate
picture of the results.
On the other hand, if the survey contains too many questions, people will
become disinterested and refuse to participate. It is hard to create a
survey that maximizes the amount of information obtained in the fewest
amount of questions. To see how Adaptiqs solves this issue, see the description
of how we provide better information with more efficiency.
Mistake #2: Too many "open-ended" questions
A second common mistake
made by beginning surveyors is to ask too many "open-ended" questions.
Open-ended questions allow people to answer a question freely without
any guidance for their response. Because respondents are allowed to write
as much as they wish, these question types can result in rich and highly
detailed data. Examples of open-ended questions are:
What is your most memorable experience of high school?
In what city were you born?
How would you prepare for a job interview?
Open ended questions have several major drawbacks. First, comparing responses
between people can be quite a challenge with open-ended questions. In
addition, providing a summary of the results of open-ended surveys require
great effort in paraphrasing the responses to capture the important concepts.
Finally, respondents will often answer open-ended questions differently
than originally intended. For this reason, open-ended survey questions
should be avoided if at all possible.
An alternative to open-ended questions is "closed-ended" questions that
provide possible responses from which to select. For example, instead
of asking "What is your favorite food?" with a place to have them enter
their responses, you would instead provide a list of options that provide
a comprehensive summary of foods such as Italian, Mexican, American, Chinese,
etc. It is important to include an exhaustive list of response options
or you run the risk of confusing or (even worse) annoying your respondents.
Because this can be time consuming and produce excessively long lists,
surveyors often provide the most common options as well as an "other"
option to allow people to insert something not listed.
Closed-ended questions allow for easy comparisons across responses and
to summarize the results of the survey. In many surveys responses can
be combined to create scales and overall "scores" that describe a particular
concept. In addition, closed-ended questions place fewer demands on respondents
as they do not have to try to guess what the surveyor is requesting.
Mistake #3: Too much attitude, not enough behavior
Another common
survey mistake is to emphasize respondent's feelings and internal reactions
as opposed to more concrete behaviors or actions.
Attitudes are difficult to measure because (1) attitudes are more likely
to be situation-specific and be influenced on the environment; (2) the
strength of an attitude can frequently change depending on a person's
mood; (2) attitudes can be sensitive to minor variations in how questions
are worded; (3) people respond as if their attitudes are long-held and
well-formed even when they were just made up on the spot; and (4) people
might not be consciously aware of their attitudes.
When the goal of the survey to describe or predict people's behavior,
stay clear of attitude questions because attitudes and behavior are not
highly correlated.
Mistake #4: Poorly worded questions
A fourth common mistake made by beginning survey surveyors is the lack
of thought in question wording. In order to avoid potential bias that
can exist in poorly worded questions, survey questions should be evaluated
using the following:
1. Do the respondents have enough information or experience to
accurately answer this question? 2. Is the question simple enough, specific enough, and sufficiently
well-defined that all of the respondents will interpret it in the same
way? 3. Does the question contain any words or phrases which could bias
respondents to answer one way over another? 4. Is it clear to respondents exactly what types of answers are
appropriate? 5. Does the question focus on a single topic or does it contain
multiple topics that should be broken up into multiple questions? 6. Are any listed response options mutually exclusive? 7. Are any assumptions implied by a question warranted?
Pretesting your questions. After the brainstorming session, ask a few qualified
subject matter experts for input on your list. Initially, you have to
carefully describe the overall purpose of the survey to your pretest group.
The more they know the survey goals, the better they will be at critiquing
your list of survey questions. Once they are familiar with the survey
goals, have them review your brainstorming list. As they review, try to
get insight on their interpretation of each question's meaning as well
as advice on improving the wording to achieve the survey goals.
In addition, ask them to create additional questions that might be missing
from the current list. Following their review, you are ready to pretest
the survey items. Pretesting survey questions will give an idea of their
performance on the survey as well as a glimpse into the overall survey
results. In addition, Adaptiqs can use the pretest data as the foundation
for its adaptive engine.
It is very important that you select people in your pretest group that
represent various segments of your overall sample population such as gender,
ethnicity, education, and other demographic variables. If the pretest
group neglects a particular group of the overall survey population, the
survey may contain biases against that group that would go undetected.
