Survey Experiments

Joe Ripberger

Experiments

  • In an experiment, the researcher randomly assigns participants to different versions of an independent variable to see how those differences affect a dependent variable
    • Independent variable (IV): what the researcher changes or manipulates; expected to influence something else
    • Dependent variable (DV): what the researcher measures to see if it changes

Note

  • In observational studies, the IV is often called an explanatory variable and the DV an outcome variable
  • In experiments, random assignment makes the IV truly independent because it is not affected by any other factors

Population-Based Survey Experiments (PBSEs)

  • Combine the strengths of experiments (causal inference through random assignment) and surveys (generalizability through representative sampling)
    • Researchers draw a sample that represents a defined population of interest
    • Participants are randomly assigned to different experimental conditions or treatments
    • Researchers then measure outcomes to estimate how each treatment affects the population

Note

  • Because of random assignment, PBSEs allow researchers to infer cause and effect
  • Because of representative sampling, they can generalize those findings to the broader population

Population-Based Survey Experiments (PBSEs)

  • Combine the internal validity of experiments with the external validity of surveys
  • Enable researchers to test causal hypotheses in realistic, policy-relevant contexts
  • Allow estimation of treatment effects that can be generalized to a larger population
  • Efficient and flexible—can be fielded online with diverse, nationally representative samples

Example

  • RQ: To what extent do associations evoked by ‘nuclear’ labels shape public perceptions of fusion energy?
  • The rest of this survey will focus on energy (electricity) production in the US. We will start with some questions about [rand_word: fusion energy | nuclear fusion | nuclear energy].
    • Can you tell us the first three words or phrases that come to you when you think about [rand_word].
      • 1st word/phrase: ________; When you think about this word or phrase, do you have positive or negative feelings?
      • 2nd word/phrase: ________; When you think about this word or phrase, do you have positive or negative feelings?
      • 3rd word/phrase: ________; When you think about this word or phrase, do you have positive or negative feelings?

Example

Some Design Considerations

  • Control vs. treatment: whether to include a true control or baseline condition
  • Pre–post vs. post-test only: whether to measure outcomes before and after treatment or only after exposure
  • Between- vs. within-subjects: whether each respondent experiences one condition or multiple conditions
  • Single vs. multiple manipulations: whether to test one manipulation or several simultaneously (factorial design)
  • With vs. without attention/manipulation checks: whether to include measures that verify respondents noticed, understood, or were influenced by the treatment
  • Direct vs. indirect outcomes: whether to measure outcomes that explicitly reference the treatment or assess downstream or implicit effects
  • Single vs. multiple outcomes: whether to focus on one primary dependent variable or assess several related outcomes
  • With vs. without covariates: whether to include covariates to improve precision or examine heterogeneous treatment effects

Common Types of Survey Experiments

  • Measurement experiments: test how question wording, ordering, or response options affect reported attitudes or behaviors
  • List (item count) experiments: measure sensitive attitudes or behaviors indirectly by asking respondents to report how many items they agree with, not which ones
  • Information provision experiments: assess how exposure to new information (facts, data, or expert statements) affects beliefs or attitudes
  • Framing experiments: manipulate the interpretive context or emphasis around an issue (e.g., public safety vs. civil liberties) to test how frames influence opinions
  • Priming experiments: make certain concepts or identities salient before measuring attitudes or judgments to test how activation affects responses
  • Vignette experiments: present a short, realistic scenario and vary specific features to examine causal effects on judgments or choices
  • Conjoint (discrete choice) experiments: ask respondents to choose between multi-attribute profiles to estimate the relative importance of each attribute
  • Contingent valuation experiments: ask respondents about their willingness to pay (or accept) for a hypothetical policy or good, often used in environmental and health economics

Measurement Experiment

  • Do you favor or oppose government welfare for the poor?
  • Do you favor or oppose government assistance for the poor?

List Experiment

  • Below is a list of things some people think make their neighborhood a worse place to live. Please tell us how many of the following you think make your neighborhood worse—do not tell us which ones.
    • Too much traffic
    • Poorly maintained public spaces
    • Rising housing costs
  • Below is a list of things some people think make their neighborhood a worse place to live. Please tell us how many of the following you think make your neighborhood worse—do not tell us which ones.
    • Too much traffic
    • Poorly maintained public spaces
    • Rising housing costs
    • Diverse families moving in from other parts of the city
  • The difference in average counts between groups estimates the share of respondents who oppose diverse families moving in from other parts of the city—without directly asking the sensitive question.

Information Provision Experiment

  • [No information]
  • Surveys of climate scientists show that over 97% of actively publishing climate scientists agree that human activities are the primary cause of recent global warming.
    • Do you think climate change is mostly caused by human activity, mostly by natural patterns, or about equally by both?

Framing Experiment

  • Many experts believe that expanding nuclear energy would reduce greenhouse gas emissions and slow climate change by providing reliable, carbon-free electricity.
  • Many experts believe that expanding nuclear energy would reduce dependence on foreign energy sources and make the U.S. more energy secure during global disruptions.
    • How much do you support or oppose building new nuclear power plants in the United States?

Priming Experiment

  • [No prime]
  • Please take a moment to indicate your race or ethnicity.
    • How much do you support or oppose affirmative action programs in college admissions?

Vignette Experiment

  • Imagine that you are at home on a Sunday at [2:30 p.m. | 11:30 p.m.]. You receive a Wireless Emergency Alert on your phone from the National Weather Service. The alert states that a tornado has been [radar-indicated | visually spotted] near your area. A warning that includes your location is in effect for the next 25 minutes. Outdoor warning sirens [are | are not] sounding.
    • How likely would you be to seek shelter immediately upon receiving this alert?

Conjoint Experiment

  • Which of the following candidates would you be more likely to vote for?
Attribute Candidate A Candidate B
Age 35 years old 55 years old
Gender Female Male
Race Black White
  • Attributes and levels
    • Age: 35 years old | 55 years old | 70 years old
    • Gender: Male | Female
    • Race: White | Black | Hispanic | Asian

Contingent Valuation Experiment

  • Imagine your community is considering a program to restore local wetlands that provide wildlife habitat and reduce flooding. If the program is approved, each household would pay a one-time fee through local taxes.
    • Would you be willing to pay [$10 | $25 | $50 | $100] as a one-time tax to fund the wetlands restoration program?

Some Challenges with Survey Experiments

  • External validity: effects observed in hypothetical survey settings may not generalize to real-world behavior
  • Demand effects: respondents may infer the experiment’s purpose and adjust their answers
  • Limited engagement: participants may skim vignettes or treatments, especially in online surveys
  • Measurement error: subtle wording or design features can influence how respondents interpret treatments or outcomes
  • Duration of effects: most experiments measure immediate responses, making it unclear how long treatment effects persist
  • Spillover effects: exposure to one treatment can influence responses to later questions or conditions (especially in within-subject designs)
  • Attrition and nonresponse: some treatments can affect completion rates, threatening balance
  • Ethical considerations: sensitive topics or deceptive framing can raise concerns about informed consent and potential harm

Exercise: Evaluate an Experiment

  • Goal: Apply key concepts from today’s readings to evaluate the design, implementation, and interpretation of a survey experiment.

  • First step: Select a study from the articles folder in the class Dropbox. Review the abstracts and identify a few that sound interesting to you. If you are already familiar with one, choose a different article to explore!

Instructions

  1. Review the study carefully
    • Identify the research question(s), type of experiment (e.g., framing, vignette, conjoint, etc.), experimental treatment(s) and outcome measure(s), and the main finding(s).
  2. Evaluate the design
    • Discuss the design’s strengths and weaknesses, considering aspects such as causal identification, validity, realism, duration, spillover, and ethics.
  3. Summarize your review and evaluation and share back with the class
    • Prepare a concise, single slide that describes the study’s design and purpose, highlights its main strengths and weaknesses with an emphasis on experimental design, and identifies 2-3 key lessons for designing strong survey experiments.