Total Survey Error (TSE) and Total Survey Quality (TSQ)

Joe Ripberger

Review

Review

  • Target population: all of the units (e.g., individuals, households, organizations) to which one desires to generalize the survey results
  • Sample (sampling) frame: the list of units in the population that the sample will be drawn from
  • Sample selection (sampling design): how the units are chosen from the sampling frame
  • Sample: all units of the population that are drawn for inclusion in the survey
  • Completed sample (respondents): all of the units sampled that complete the survey questionnaire
  • Survey mode (mode of data collection): the way questions are delivered to respondents and how their answers are returned to the researcher

Review

Assignment

Assignment

Assessing Public Interpretations of Risk Categories in Spanish
Who was the target population for the study? - Selected Choice What was the sample frame for the study? - Selected Choice What was the sampling design for the study? - Selected Choice
All adults in a country (please list the country) Social media platform or email distribution list Multi-stage sample (combination of the above methods in steps)
All adults in a country (please list the country) Address-based list of households Simple random sample (every unit had an equal chance of selection)
A specific demographic or social group (please list the group) Online research panel maintained by a survey firm Other method (please describe)
A specific demographic or social group (please list the group) Online research panel maintained by a survey firm Non-probability sample (e.g., convenience, quota, snowball, purposive)
Another group (please describe) Online research panel maintained by a survey firm Non-probability sample (e.g., convenience, quota, snowball, purposive)
A specific demographic or social group (please list the group) Online research panel maintained by a survey firm Non-probability sample (e.g., convenience, quota, snowball, purposive)
All adults in a country (please list the country) Online research panel maintained by a survey firm Non-probability sample (e.g., convenience, quota, snowball, purposive)
A specific demographic or social group (please list the group) Online research panel maintained by a survey firm Non-probability sample (e.g., convenience, quota, snowball, purposive)
A specific demographic or social group (please list the group) Online research panel maintained by a survey firm Non-probability sample (e.g., convenience, quota, snowball, purposive)

Assignment

Assessing Public Interpretations of Risk Categories in Spanish
How many people (or units) were selected for inclusion in the study (the sample)? - Number reported (please enter below) - Text How many people (or units) actually completed the survey (the respondents)? - Number reported (please enter below) - Text What was the survey mode for this study? - Selected Choice
NA NA Mixed mode (more than one of the above, please describe)
NA NA Telephone survey (automated / robocall)
1307 1050 Web / online questionnaire
1307 1050 Web / online questionnaire
1307 1050 Web / online questionnaire
1307 1050 Web / online questionnaire
1307 1050 Web / online questionnaire
1307 1050 Web / online questionnaire
1050 1307 Web / online questionnaire

Assignment

Is Sexism for White People
Who was the target population for the study? - Selected Choice What was the sample frame for the study? - Selected Choice What was the sampling design for the study? - Selected Choice
All adults in a country (please list the country) Another list or source (please describe) Non-probability sample (e.g., convenience, quota, snowball, purposive)
A specific demographic or social group (please list the group) Another list or source (please describe) Stratified random sample (population divided into groups, then sampled within each)
A specific demographic or social group (please list the group) Online research panel maintained by a survey firm Cluster sample (groups or clusters selected first, then individuals within clusters)
Another group (please describe) Another list or source (please describe) Non-probability sample (e.g., convenience, quota, snowball, purposive)
A specific demographic or social group (please list the group) Online research panel maintained by a survey firm Non-probability sample (e.g., convenience, quota, snowball, purposive)
All adults in a country (please list the country) Another list or source (please describe) Simple random sample (every unit had an equal chance of selection)
A specific demographic or social group (please list the group) Another list or source (please describe) Multi-stage sample (combination of the above methods in steps)
Residents of a specific region, state, or city (please list the region, state, or city) Another list or source (please describe) Stratified random sample (population divided into groups, then sampled within each)
Residents of a specific region, state, or city (please list the region, state, or city) Voter registration list or electoral roll Non-probability sample (e.g., convenience, quota, snowball, purposive)

Assignment

Is Sexism for White People
How many people (or units) were selected for inclusion in the study (the sample)? - Number reported (please enter below) - Text How many people (or units) actually completed the survey (the respondents)? - Number reported (please enter below) - Text What was the survey mode for this study? - Selected Choice
NA 13188 Mixed mode (more than one of the above, please describe)
11861 1327 Mixed mode (more than one of the above, please describe)
11861 NA Web / online questionnaire
NA 13188 Mixed mode (more than one of the above, please describe)
NA 11861 Web / online questionnaire
NA 13188 Mixed mode (more than one of the above, please describe)
NA 132711861 Web / online questionnaire
NA 1327 Mixed mode (more than one of the above, please describe)
NA 1327 Other mode (please describe)

Total Survey Error (TSE)

Early Notions of Survey Error

  • Here are the 2024 Presidential General Election results for Cleveland County, Oklahoma, based on county-level data from the statewide breakdown:
    • Donald Trump (R): 67,225 votes (56.35%)
    • Kamala Harris (D): 49,432 votes (41.44%)
    • Other candidates: 2,637 votes (2.21%)
    • Total ballots cast: 119,294
  • Imagine that we were unable to access this full dataset, and were only able to observe votes from a random sample of 500 people…

Early Notions of Survey Error

sample_n(elec_data, 500)
# A tibble: 500 × 6
   voter_id state    county            year election             vote  
      <dbl> <chr>    <chr>            <dbl> <chr>                <chr> 
 1    32865 Oklahoma Cleveland County  2024 US President General Harris
 2   118220 Oklahoma Cleveland County  2024 US President General Harris
 3   109760 Oklahoma Cleveland County  2024 US President General Harris
 4    70245 Oklahoma Cleveland County  2024 US President General Harris
 5    94076 Oklahoma Cleveland County  2024 US President General Trump 
 6    61864 Oklahoma Cleveland County  2024 US President General Trump 
 7    78848 Oklahoma Cleveland County  2024 US President General Harris
 8    94915 Oklahoma Cleveland County  2024 US President General Trump 
 9   106099 Oklahoma Cleveland County  2024 US President General Trump 
10     9670 Oklahoma Cleveland County  2024 US President General Trump 
# ℹ 490 more rows

Early Notions of Survey Error

set.seed(1); sample_n(elec_data, 500) |> count(vote) |> mutate(p = n / sum(n))
# A tibble: 3 × 3
  vote       n     p
  <chr>  <int> <dbl>
1 Harris   199 0.398
2 Other     11 0.022
3 Trump    290 0.58 
set.seed(2); sample_n(elec_data, 500) |> count(vote) |> mutate(p = n / sum(n))
# A tibble: 3 × 3
  vote       n     p
  <chr>  <int> <dbl>
1 Harris   219 0.438
2 Other      7 0.014
3 Trump    274 0.548
set.seed(3); sample_n(elec_data, 500) |> count(vote) |> mutate(p = n / sum(n))
# A tibble: 3 × 3
  vote       n     p
  <chr>  <int> <dbl>
1 Harris   214 0.428
2 Other      7 0.014
3 Trump    279 0.558

Early Notions of Survey Error

  • Sampling variability: natural variation in survey estimates because we observe only a sample of the population, not everyone
    • Different random samples = different estimates (sometimes higher, sometimes lower)
  • Early approaches to survey error focused on this form of error (sampling errors)
    • Tools: standard error (SE), confidence interval (CI), margin of error (MoE)
    • Standard error (SE): \(SE(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\)
    • Margin of error (MoE): \(MoE = z_{\alpha/2} \times SE(\hat{p})\)
    • Confidence interval (CI): \(CI = \hat{p} \;\pm\; MoE\)

Early Notions of Survey Error

set.seed(3); sample_n(elec_data, 500) |> count(vote) |> mutate(p = n / sum(n))
# A tibble: 3 × 3
  vote       n     p
  <chr>  <int> <dbl>
1 Harris   214 0.428
2 Other      7 0.014
3 Trump    279 0.558
  • Standard Error (SE): \(\sqrt{\frac{0.558(1-0.558)}{500}} = 0.022\)
  • Margin of Error (MoE): \(1.96 \times 0.022 = 0.043 \; (4.3\%)\)
  • Confidence Interval (CI): \(0.558 \;\pm\; 0.043 = [0.515,\; 0.601]\)
  • Interpretation: Trump support is estimated at 55.8%, with a margin of error of ±4.3 percentage points and a 95% CI of 51.5%–60.1%.

Total Survey Error (TSE)

  • Total survey error (TSE): accumulation of all errors that may arise in the design, collection, processing, and analysis of survey data
    • Recognizes multiple sources of error beyond sampling errror
  • Broadly, two main types of error:
    • Sampling errors: errors that arise from sampling
    • Non-sampling errors: errors that arise from everything else—design, data collection, processing procedures, and analysis

Measuring Survey Errors

  • Survey accuracy: depends on two components:
    • Variance: variability of estimates across different samples (e.g., different random samples give slightly different results)
    • Bias: systematic deviation of an estimate from the true value (e.g., consistent undercounting of renters in a survey frame)

Measuring Survey Errors

  • Total survey error (TSE) ≈ Bias² + Variance
    • Links survey methodology to a classic measure in statistics, the mean squared error (MSE)
    • \(\text{MSE}(\hat{\theta}) = \mathbb{E}\!\left[(\hat{\theta} - \theta)^2\right] = \operatorname{Var}(\hat{\theta}) + \big(\operatorname{Bias}(\hat{\theta})\big)^2\)

Measuring Survey Errors

  • Suppose you estimate average household income with an online opt-in survey
    • Variance: If you repeated the survey with another sample from the same panel, the mean might bounce around (because the sample is small)
    • Bias: If your survey systematically underrepresents low-income households, your estimate will be too high on average
  • The MSE captures both problems at once: how much error you should expect in your estimate
  • Challenge:
    • Variance can be estimated from the sample (e.g., standard error, confidence interval)
    • Bias is usually unobservable unless the true population value is known

Sources of Survey Errors

Coverage Error

  • Coverage error: difference between the target population and the sampling frame; occurs when some population units have no chance of selection or are erroneously included
    • Example: a telephone survey that misses cell-phone–only households
    • Other examples?

Sampling Error

  • Sampling error: difference between an estimate based on a sample and the true population value that arises because only a subset of units is observed
    • Example: two random samples of households give slightly different unemployment rates
    • Other examples?

Nonresponse Error

  • Nonresponse error: difference between respondents and nonrespondents that leads to estimates not representing the intended sample or population
    • Example: people with very low incomes are less likely to respond to surveys (unit nonresponse)
    • Example: people with very high incomes are less likely to respond to income questions (item nonresponse)
    • Other examples?

Adjustment Error

  • Adjustment error: difference between the adjusted (e.g., weighted or imputed) survey estimates and the true population values that arises when the adjustment procedures are misspecified or imperfect
    • Example: calibrating to an external benchmark (e.g., voter turnout) when that benchmark itself is measured with error
    • Other examples?

Specification (Validity) Error

  • Specification (validity) error: difference between the concept the researcher intends to measure and the construct actually captured by the survey question
    • Example: using job satisfaction as a proxy for overall life satisfaction
    • Other examples?

Measurement Error

  • Measurement error: difference between the value a survey question records and the true value for the respondent, due to question wording, interviewer effects, mode, recall, or response biases
    • Example: underreporting alcohol consumption or overreporting voting (social-desirability bias)
    • Other examples?

Processing Error

  • Processing error: difference introduced during data handling, such as coding, keying, editing, or weighting, that causes the stored data to deviate from the respondent’s intended answer
    • Example: person in charge of data entry records 5 instead of 50
    • Other examples?

Total Survey Error (TSE)

Error Source Short Definition Example
Coverage error Frame misses some units (or includes extras) Cell-only households omitted from phone survey
Sampling error Estimate differs by chance from sample to sample Two random samples give different unemployment rates
Nonresponse error Respondents ≠ nonrespondents Low-income households less likely to respond
Adjustment error Corrections distort estimates Weights based on old census data
Specification (validity) Measure doesn’t match intended concept Using job satisfaction to measure life satisfaction
Measurement error Response differs from true value Overreporting voting; underreporting alcohol
Processing error Mistakes during data handling Income typed as 5,000 instead of 50,000

Total Survey Quality (TSQ)

Total Survey Quality (TSQ)

  • Shift in perspective
    • TSE → focus on minimizing error
    • TSQ → balance error, cost, and timeliness to optimize overall quality,
      given constraints
  • Fit for purpose
    • The appropriate level of quality depends on the intended use of the data
    • Goal is not zero error, but sufficient quality for decision-making

Total Survey Quality (TSQ)

  • Accuracy: total survey error is minimized
  • Credibility: data are considered trustworthy by the survey community
  • Comparability: demographic, spatial, and temporal comparisons are valid
  • Usability/interpretability: documentation is clear and metadata are well-managed
  • Relevance: data satisfy users needs
  • Accessibility: access to the data is user friendly
  • Timeliness/punctuality: data deliveries adhere to schedules
  • Completeness: data are rich enough to satisfy the analysis objectives without undue burden on respondents
  • Coherence: estimates from different sources can be reliably combined

Using TSE and TSQ

  • TSE and TSQ are often applied to multiple phases of the survey process:
    1. Survey design
    2. Survey administration
    3. Survey evaluation

Paradata

  • Paradata: auxiliary data about the survey process that can be used to assess and improve survey quality

Activity

Instructions

  1. Choose a survey topic (keep it simple; e.g., student quality of life)
  2. Sketch your survey design
    • Define the target population, sampling approach (sample frame and selection), and mode of data collection
  3. Diagnose errors (TSE)
    • Identify 2–3 potential sources of error
    • Discuss how you would minimize each in practice
  4. Enhance quality (TSQ)
    • Identify 2–3 elements of survey quality
    • Describe what design choices help achieve them
  5. Prepare a quick report-out (3 minutes per group)
    • Name of your survey
    • Key errors minimized
    • Quality elements maximized

Groups

set.seed(5000)
data |>
  distinct(name) |>
  slice_sample(prop = 1) |>
  mutate(group = ceiling(row_number() / 3)) |> 
  group_by(group) |>
  summarise(names = paste(name, collapse = ", "), .groups = "drop")
# A tibble: 5 × 2
  group names                                         
  <dbl> <chr>                                         
1     1 Ben Fellman, Bulbul Ahmed, Joy Rhodes         
2     2 Alexis Jones, Laken DeBoard, Vanessa Dunham   
3     3 Nathaniel Reeves, Anna Wanless, Charles Kenney
4     4 Elizabeth Meister, Lauren Harvey, Renata Emini
5     5 Riley Worley, Abby Bitterman

Instructions

  1. Choose a survey topic (keep it simple; e.g., student quality of life)
  2. Sketch your survey design
    • Define the target population, sampling approach (sample frame and selection), and mode of data collection
  3. Diagnose errors (TSE)
    • Identify 2–3 potential sources of error
    • Discuss how you would minimize each in practice
  4. Enhance quality (TSQ)
    • Identify 2–3 elements of survey quality
    • Describe what design choices help achieve them
  5. Prepare a quick report-out (3 minutes per group)
    • Name of your survey
    • Key errors minimized
    • Quality elements maximized
45:00

Report-out

03:00

Total Survey Error (TSE) and Total Survey Quality (TSQ)