Mode effects on survey item measurement: A systematic review of the experimental evidence

Survey data are increasingly collected using mixed-mode designs. However, the measurement of survey items may differ across modes, introducing ‘mode effects’, a type of systematic measurement error which can bias analyses of mixed-mode data. While the theoretical mechanisms giving rise to mode effects have been discussed in detail, the empirical evidence on their occurrence and size is fragmented. In addition, while many existing statistical approaches for handling mode effects require unrealistic assumptions, other more suitable approaches remain underutilised due to the need for external evidence on the magnitude of mode effects. To address this, we conducted a systematic review of the experimental literature on mode effects. We searched multiple bibliographic databases, grey literature sources, and implemented backwards and forwards citation screening. Studies eligible for inclusion were (quasi-)experimental, sampled from the general population (or age-, sex-, region-specific strata), and reported mode effect estimates on item measurement. We extracted comprehensive information relating to the study design, sampling, mode effect estimates, and reporting. Ninety experimental studies published between 1967 and 2024 met the inclusion criteria, which included 4,113 mode effect estimates for 3,545 unique variables in total. Mode effects were generally small, typically below 0.2 SD. However, larger mode effects were more commonly observed when modes differed by interviewer involvement or by question delivery (visual vs aural), as well as for sensitive items (e.g., sexual behaviour, social life), which aligns with pre-existing theory on the causes of mode effects. Generally, where mode effects occur, they are item-, mode-, and population-specific. Reporting quality varied substantially and insufficient details regarding randomisation compliance, non- response, and uncertainty of estimates were common. We collated all mode effect estimates into a free online database and provide a set of recommendations to improve the reporting of future studies.