When, and how much, might effect estimates be biased when analysing mixed-mode survey data? The roles of mode effects, mode selection, and mode split

Survey data are increasingly collected using mixed-mode designs (e.g. personal interview and web questionnaire). Little is known, however, about the extent to which this introduces bias in subsequent analyses, or whether simply including mode as a covariate addresses it. Using data simulations, we identified the conditions under which mode effects (mode measurement differences) and mode selection (respondent differences) introduce bias, complementing this with empirical illustrations using mixed-mode data from the 1958 National Child Development Study. In simulations, absent mode selection, substantial bias arose only from unusually large mode effects, but was amplified when mode selection was present. Controlling for mode under strong mode selection introduced substantial bias, including sign-reversal of the estimate. Bias was more pronounced when the sample was split equally between modes. The direction and size of bias will depend on the direction of all effects. In our empirical illustration, the results were largely unchanged after controlling for mode, possibly reflecting weaker mode selection and mode effects in these data. These findings highlight that, while possible, substantial bias from mixed-mode designs may be unlikely in many practical settings. However, the risk of bias, and the appropriate strategy to address it, should be considered on an analysis-specific basis.