Experimental Evidence on the Effect of Mixed-Mode Survey Designs on Item Measurement and Variable Correlations

Background: Mixed-mode designs can generate bias in all types of analysis of survey data due to systematic differences in how survey participants may respond according to mode – a phenomenon referred to as a mode effect. There is widespread evidence of mode effects upon item measurement (e.g., item means and proportions), but less on associations between survey items, despite estimating associations being a primary use of survey data. Methods: We estimated average differences (means and proportions) in item responses according to mode using data from an experiment in which participants were randomized to respond via web questionnaire, video interview, or in-person interview. We then used these estimates to simulate (counterfactual) data from two web-then-in-person sequential mixed-mode surveys of young adults in the United Kingdom (Next Steps and the Millennium Cohort Study, MCS) as if they were collected using a single mode, web. We estimated associations between variables using this simulated data and compared these against associations obtained using the observed (i.e., factual) data; the difference between the two provides a measure of variation in variable correlations reflecting the mixed-mode survey design. Results: Mode effects were observed for an array of items comparing web with in-person and video modes; differences between video and in-person were more muted. Participants in web were more likely to report negatively valent values for sensitive items, in particular (e.g., poorer physical and mental health, lower life satisfaction, and greater financial difficulty), though effect sizes were generally small with few exceeding 0.2 SD and all below 0.3 SD. The impact of mode effects on variable associations in the mixed-mode surveys was also generally small: a 0.004 SD change in absolute effect sizes on average in both Next Steps and the MCS. Conclusion: Items related to sensitive characteristics showed small mode effects when comparing web and in-person or video interview modes. Mode effects did not appear to translate into substantial changes to associations between variables as assessed in a sequential web-then-in-person mixed-mode survey compared with a counterfactual web-only survey.