Questionnaire Design and Testing

Joe Ripberger

Questionnaire Design (Wording)

Survey Question Commandments

1. Thou shalt not write a question that thou thyself cannot answer.
   • Write only questions you could confidently and accurately answer yourself.
2. Thou shalt ask no question for which thou knowest not the purpose of the answer.
   • Include only questions whose answers will serve a defined analytic or decision-making purpose.

Nonsensitive Behavioral Questions (Groves et al. 2009)

1. With closed questions, include all reasonable possibilities as explicit response options.
2. Make the questions as specific as possible.
3. Use words that virtually all respondents will understand.
4. Lengthen the questions by adding memory cues to improve recall.
5. When forgetting is likely, use aided recall.
6. When the events of interest are frequent but not very involving, have respondents keep a diary.
7. When long recall periods must be used, use a life event calendar to improve reporting.
8. To reduce telescoping errors, ask respondents to use household records or use bounded recall (or do both).
9. If cost is a factor, consider whether proxies might be able to provide accurate information.

Sensitive Behavioral Questions (Groves et al. 2009)

1. Use open rather than closed questions for eliciting the frequency of sensitive behaviors.
2. Use long rather than short questions.
3. Use familiar words in describing sensitive behaviors.
4. Deliberately load the question to reduce misreporting.
5. Ask about long periods (such as one’s entire lifetime) or periods from the distant past first in asking about sensitive behaviors.
6. Embed the sensitive question among other sensitive items to make it stand out less.
7. Use self-administration or some similar method to improve reporting.
8. Consider collecting the data in a diary.
9. At the end of the questionnaire, include some items to assess how sensitive the key behavioral questions were.
10. Collect validation data.

Attitude Questions (Groves et al. 2009)

1. Specify the attitude object clearly.
2. Avoid double-barreled questions.
3. Measure the strength of the attitude, if necessary using separate items for this purpose.
4. Use bipolar items except when they might miss key information.
5. The alternatives mentioned in the question have a big impact on the answers; carefully consider which alternatives to include.
6. In measuring change over time, ask the same questions each time.
7. When asking general and specific questions about a topic, ask the general question first.
8. When asking questions about multiple items, start with the least popular one.
9. Use closed questions for measuring attitudes.
10. Use five-to seven-point response scales and label every scale point.
11. Start with the end of the scale that is the least popular.
12. Use analogue devices (such as thermometers) to collect more detailed scale information.
13. Use ranking only if the respondents can see all the alternatives; otherwise, use paired comparisons.
14. Get ratings for every item of interest; do not use check-all-that-apply items.

Conventional Wisdom (Krosnick and Presser 2010)

1. Use simple, familiar words (avoid technical terms, jargon, and slang.
2. Use simple syntax.
3. Avoid words with ambiguous meanings, i.e., aim for wording that all respondents will interpret in the same way.
   
4. Strive for wording that is specific and concrete (as opposed to general and abstract).
5. Make response options exhaustive and mutually exclusive.
6. Avoid leading or loaded questions that push respondents toward an answer.
   
7. Ask about one thing at a time (avoid double-barreled questions).
   
8. Avoid questions with single or double negations.

Question Order (Krosnick and Presser 2010)

1. Early questions should be easy and pleasant to answer, and should build rapport between the respondent and the researcher.
   
2. Questions at the very beginning of a questionnaire should explicitly address the topic of the survey, as it was described to the respondent prior to the interview.
   
3. Questions on the same topic should be grouped together.
   
4. Questions on the same topic should proceed from general to specific.
   
5. Questions on sensitive topics that might make respondents uncomfortable should be placed at the end of the questionnaire.
6. Filter questions should be included, to avoid asking respondents questions that do not apply to them.

Open-Ended Questions (Dillman et al. 2014)

1. Specify the type of response desired in the question stem.
2. Avoid making respondents (or interviewers) calculate sums; when possible, have the computer do it.
3. Provide extra motivation to respond.
4. Use nondirective probes to obtain more information on open-ended items.

All Closed-Ended Questions (Dillman et al. 2014)

1. When asking either/or types of questions, state both the positive and negative side in the question stem.
2. Develop lists of answer categories that include all reasonable possible answers.
3. Develop lists of answer categories that are mutually exclusive.
4. Consider what types of answer spaces are most appropriate for the measurement intent.

Nominal Closed-Ended Questions (Dillman et al. 2014)

1. Ask respondents to rank only a few items at once rather than a long list.
2. Avoid bias from unequal comparisons.
3. Randomize response options if there is concern about order effects.
4. Use forced-choice questions instead of check-all-that-apply questions.

Ordinal Closed-Ended Questions (Dillman et al. 2014)

1. Choose between a unipolar or a bipolar scale.
2. Choose an appropriate scale length—in general, limit scales to four or five categories.
3. Choose direct or construct-specific labels to improve cognition.
4. If there is a natural metric (e.g., frequencies, amounts, sizes, etc.), use it instead of vague quantifiers.
5. Provide balanced scales where categories are relatively equal distances apart conceptually.
6. Verbally label all categories.
7. Remove numeric labels from vague quantifier scales whenever possible.
8. Consider branching (or decomposing) bipolar scales to ease respondent burden and improve data quality.
9. Provide scales that approximate the actual distribution of the characteristic in the population, or ask the question in an open-ended format to avoid biasing responses.

Summary Points (Schaeffer and Dykema 2020)

1. Given the continuing confirmations that verbal labels increase reliability and the need for consistency across modes and presentations, five categories for unipolar and possibly bipolar scales is both justifiable and practical at this time.
2. Studies that scale adverbs and phrases support the categories “not at all, slightly/a little, somewhat/moderately, very, extremely” for unipolar intensity questions; “never, rarely, sometimes, very often, extremely often/always” for relative frequency questions; and “none, a little/a little bit, some/somewhat, quite a bit, a great deal” for quantity-based questions. The slashes indicate alternatives with similar scale values.
   
3. For measures of subjective concepts, the current recommendation to use item-specific response formats rather than agree-disagree response formats is based on strong theoretical reasons, the ease with which respondents understand the questions, and available evidence on data quality.
   
4. Yes–no checklists in which items are presented as a grid (or battery) that requests a “yes” or “no” answer for each item are recommended over CATA formats.
   
5. Questions should be written so that the format for the response projected by the question matches.

Questionnaire Design (Design)

Self-Administered Questions (Groves et al. 2009)

1. Use visual elements in a consistent way to define the desired path through the questionnaire.
   
2. When the questionnaire must change its conventions part way through, prominent visual guides should alert respondents to the switch.
   
3. Place directions where they are to be used and where they can be seen.
   
4. Present information that needs to be used together in the same location.
   
5. Ask one question at a time.

General Guidelines (Dillman et al. 2014)

1. Use darker and/or larger print for the question stem and lighter and/or smaller print for answer choices and answer spaces.
   
2. Use spacing to help create subgrouping within a question.
   
3. Visually standardize all answer spaces or response options.
   
4. Use visual design properties to emphasize elements that are important to the respondent and to deemphasize those that are not.
   
5. Choose font, font size, and line length to ensure the legibility of the text.
   
6. Integrate special instructions into the question where they will be used, rather than including them as free-standing entities.
   
7. Separate optional or occasionally needed instructions from the question stem by font or symbol variation.

Open-Ended Questions (Dillman et al. 2014)

1. Provide a single answer box if only one answer is needed and multiple answer boxes if multiple answers are needed.
   
2. Provide answer spaces that are sized appropriately for the response task.
   
3. To encourage the use of proper units or a desired response format, provide labels and templates with answer spaces.

Closed-Ended Questions (Dillman et al. 2014)

1. Align response options vertically in one column or horizontally in one row, and provide equal distance between categories.
   
2. Place nonsubstantive options after and separate from substantive options.
   
3. Consider using differently shaped answer spaces (circles and squares) to help respondents distinguish between single- and multiple-answer questions.

Pages or Screens (Dillman et al. 2014)

1. Establish grouping and subgrouping within and across questions in the questionnaire.
   
2. Establish consistency in the visual presentation of questions, and use alignment and vertical spacing to help respondents organize the information on the page.
   
3. Use color and contrast to help respondents recognize the components of the questions and the navigational path through the questionnaire.
   
4. Visually group related information in regions through the use of contrast and enclosure.
   
5. Consistently identify the beginning of each question and/or section.
   
6. Use visual elements and properties consistently across questions and pages/screens to visually emphasize or deemphasize certain types of information.
   
7. Avoid visual clutter.
   
8. Avoid placing questions side by side on a page so that respondents are not asked to answer two questions at once.
   
9. Minimize the use of matrices and grids, and when they cannot be avoided, minimize their complexity.

Questionnaire Testing (Evaluation)

Questionnaire Testing (Evaluation)

• The objective of questionnaire testing is to ensure that survey questions measure what they are intended to measure, that respondents understand and can answer them as intended, and that they can be administered effectively under real field conditions.

• Common methods include:

  • Expert reviews
    
  • Focus groups
    
  • Cognitive testing (cognitive interviews)
  • Field pretests
  • Split-ballot experiments

Expert Reviews

• A structured review of draft questions by subject-matter or questionnaire design experts
• Used to assess whether the questions:
  • Accurately measure the intended concepts
    
  • Are appropriate for the target population
    
  • Meet standards for clarity, neutrality, and completeness
    
• Usually conducted early in questionnaire development
  
• Helps identify conceptual gaps, poor wording, or leading phrasing before field testing

Focus Groups

• Semi-structured discussions with members of the target population
• Used to understand how respondents:
  • Think and talk about key issues
    
  • Interpret terms and concepts
    
  • Frame their opinions or experiences in natural language
    
• Conducted before finalizing question wording
  
• Informs both content validity and the vocabulary used in survey items

Cognitive Testing (Cognitive Interviews)

• One-on-one interviews where respondents answer draft survey questions while explaining their thought process
• Used to understand how respondents:
  • Interpret the question
    
  • Retrieve relevant information from memory
    
  • Formulate and report an answer
    
• Reveals cognitive difficulties, ambiguous wording, and potential response errors
  
• Provides direct insight into how respondents actually process the question

Field Pretests

• Small-scale version of the survey, usually fewer than 100 interviews, conducted under realistic field conditions
  
• Used to test:
  • Interviewer performance and instructions
  • Sampling and administration procedures
  • Question comprehension and flow
• Data may be tabulated to identify problematic items
  
• Interviewers may be debriefed to discuss respondent difficulties
  
• Recordings can be behavior-coded to quantify comprehension problems

Split-Ballot Experiments

• Experimental design where different subgroups of the pretest sample receive alternative versions of the same question
• Used to estimate how different wordings or formats affect responses
• Allows researchers to identify measurement bias or context effects
• Provides empirical evidence about which question version performs best
• Often used after pretesting to confirm the most valid and reliable item