Workshop

Joe Ripberger

Questionnaire Testing (Evaluation)

• The objective of questionnaire testing is to ensure that survey questions measure what they are intended to measure, that respondents understand and can answer them as intended, and that they can be administered effectively under real field conditions.

• Common methods include:

  • Expert reviews
  • Focus groups
  • Cognitive testing (cognitive interviews)
  • Field pretests
  • Split-ballot experiments

Expert Reviews

• A structured review of draft questions by subject-matter or questionnaire design experts
• Used to assess whether the questions:
  • Accurately measure the intended concepts
    
  • Are appropriate for the target population
    
  • Meet standards for clarity, neutrality, and completeness
    
• Usually conducted early in questionnaire development
  
• Helps identify conceptual gaps, poor wording, or leading phrasing before field testing

Cognitive Testing (Cognitive Interviews)

• One-on-one interviews where respondents answer draft survey questions while explaining their thought process
• Used to understand how respondents:
  • Interpret the question
    
  • Retrieve relevant information from memory
    
  • Formulate and report an answer
    
• Reveals cognitive difficulties, ambiguous wording, and potential response errors
  
• Provides direct insight into how respondents actually process the question

Field Pretests

• Small-scale version of the survey, usually fewer than 100 interviews, conducted under realistic field conditions
  
• Used to test:
  • Interviewer performance and instructions
  • Sampling and administration procedures
  • Question comprehension and flow
• Data may be tabulated to identify problematic items
  
• Interviewers may be debriefed to discuss respondent difficulties
  
• Recordings can be behavior-coded to quantify comprehension problems

Instructions

• Expert Review: solicit feedback from your assigned expert reviewer(s).
  • Expert reviewers should:
    • Evaluate each survey question for clarity, relevance, and potential bias.
      
    • Identify confusing wording, missing response options, or structural issues.
      
    • Provide written comments and suggested revisions.
    • Use the class cheat sheet if it helps!
• Cognitive Test: conduct a one-on-one (or small group) cognitive interview with your assigned participant(s).
  • Cognitive interviewees should:
    • Answer the survey questions aloud.
      
    • Describe how they interpret each question and how they arrive at their responses.
      
    • Flag any wording that seems unclear, misleading, or difficult to answer.
• Field pretests: Program your survey in Qualtrics (if you haven’t already).
  • Test the full survey yourself.
    
  • Ask at least 10 additional people (friends, family, etc.) to complete the survey and provide feedback on clarity, flow, and any technical issues.

Assignments

Survey Developer(s)	Expert Reviewers	Cognitive Interviewee(s)
Alexis	Lauren, Laken	Ben
Ben, Lauren, Vanessa	Joy, Bulbul	Charlie, Nate
Bulbul	Riley, Charlie	Joy
Charlie	Elizabeth	Abby, Vanessa
Joy, Abby	Vanessa	Riley
Laken	Ben	Elizabeth, Alexis
Nate	Alexis	Lauren, Bulbul
Riley, Anna, Elizabeth	Abby, Nate	Laken

Presentation Instructions (December 8)

Prepare a 20-minute presentation that explains your survey design from research question to planned analysis.

• Research question / objective
  • State your core research question(s) and why they matter
    
  • Identify the key concepts or constructs you intend to measure
• Survey design plan
  • Define your target population
  • Describe your sampling strategy and data-collection mode
    
  • Note anticipated challenges and how you plan to address them
• Questionnaire design
  • Present the structure and flow of your questionnaire
    
  • Show examples of key items
  • Explain steps taken to test the questionnaire

Presentation Instructions (December 8)

• Survey experiment
  • Describe your experimental question or hypothesis
    
  • Define conditions/treatments and explain randomization
    
  • Explain what outcome(s) you plan to compare
• Planned analysis
  • Outline your approach to cleaning and preparing the data
    
  • Describe your planned weighting strategy
    
  • Identify the tables, figures, comparisons, or statistics you plan to produce

Final Deliverable Instructions (December 19)

• Imagine that a foundation is considering providing support for your survey research project. Before committing funds, they ask for a short proposal that clearly explains what you plan to study, who you plan to survey, how you will design your questionnaire and experiment, and how you intend to analyze the data once it is collected. They don’t need a full grant application—just a concise, well-reasoned description of your survey design that demonstrates you understand the methodological choices involved.

• Your assignment mirrors that request: prepare a crisp, professional proposal that addresses each of the following components, using the suggested word counts as a guide:

  • Research question / objective: 250–300 words
  • Survey design plan: 500–700 words
  • Questionnaire design: 400–600 words
  • Survey experiment: 400–600 words
  • Planned analysis: 400–600 words

• Include the full draft survey as an appendix.

Tips

• The final presentation and deliverable are opportunities to practice explaining how the concepts from the course fit together.
  • Use appropriate technical vocabulary
    
  • Reference ideas and principles from our readings and discussions
    
  • Communicate your reasoning clearly and professionally
• I want to understand your project clearly, and I also want to see how you are applying the concepts you’ve learned in this class.