IDEAS home Printed from https://ideas.repec.org/p/osf/socarx/bc7qn_v1.html

Mode effects on survey item measurement: A systematic review of the experimental evidence

Author

Listed:
  • Tomova, Georgia D
  • Silverwood, Richard J.
  • Wright, Liam

Abstract

Survey data are increasingly collected using mixed-mode designs. However, the measurement of survey items may differ across modes, introducing ‘mode effects’, a type of systematic measurement error which can bias analyses of mixed-mode data. While the theoretical mechanisms giving rise to mode effects have been discussed in detail, the empirical evidence on their occurrence and size is fragmented. In addition, while many existing statistical approaches for handling mode effects require unrealistic assumptions, other more suitable approaches remain underutilised due to the need for external evidence on the magnitude of mode effects. To address this, we conducted a systematic review of the experimental literature on mode effects. We searched multiple bibliographic databases, grey literature sources, and implemented backwards and forwards citation screening. Studies eligible for inclusion were (quasi-)experimental, sampled from the general population (or age-, sex-, region-specific strata), and reported mode effect estimates on item measurement. We extracted comprehensive information relating to the study design, sampling, mode effect estimates, and reporting. Ninety experimental studies published between 1967 and 2024 met the inclusion criteria, which included 4,113 mode effect estimates for 3,545 unique variables in total. Mode effects were generally small, typically below 0.2 SD. However, larger mode effects were more commonly observed when modes differed by interviewer involvement or by question delivery (visual vs aural), as well as for sensitive items (e.g., sexual behaviour, social life), which aligns with pre-existing theory on the causes of mode effects. Generally, where mode effects occur, they are item-, mode-, and population-specific. Reporting quality varied substantially and insufficient details regarding randomisation compliance, non- response, and uncertainty of estimates were common. We collated all mode effect estimates into a free online database and provide a set of recommendations to improve the reporting of future studies.

Suggested Citation

  • Tomova, Georgia D & Silverwood, Richard J. & Wright, Liam, 2026. "Mode effects on survey item measurement: A systematic review of the experimental evidence," SocArXiv bc7qn_v1, Center for Open Science.
    • Handle: RePEc:osf:socarx:bc7qn_v1
    DOI: 10.31219/osf.io/bc7qn_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/695d48d55d51b4379906c45a/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/bc7qn_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Almut Schumann & Detlev Lück, 2023. "Better to ask online when it concerns intimate relationships? Survey mode differences in the assessment of relationship quality," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 48(22), pages 609-640.
    2. Lena Wettergren & Elisabet Mattsson & Louise von Essen, 2011. "Mode of administration only has a small effect on data quality and self‐reported health status and emotional distress among Swedish adolescents and young adults," Journal of Clinical Nursing, John Wiley & Sons, vol. 20(11‐12), pages 1568-1577, June.
    3. Giorgio Piccitto & Aart C. Liefbroer & Tom Emery, 2022. "Does the Survey Mode Affect the Association Between Subjective Well-being and its Determinants? An Experimental Comparison Between Face-to-Face and Web Mode," Journal of Happiness Studies, Springer, vol. 23(7), pages 3441-3461, October.
    4. Mingnan Liu & Frederick G. Conrad & Sunghee Lee, 2017. "Comparing acquiescent and extreme response styles in face-to-face and web surveys," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(2), pages 941-958, March.
    5. Brooke Helppie-McFall & Joanne W. Hsu, 2017. "A test of web and mail mode effects in a financially sensitive survey of older Americans," Journal of Economic and Social Measurement, IOS Press, issue 2, pages 151-169.
    6. Annette Jäckle & Caroline Roberts & Peter Lynn, 2010. "Assessing the Effect of Data Collection Mode on Measurement," International Statistical Review, International Statistical Institute, vol. 78(1), pages 3-20, April.
    7. Tellis, Gerard J. & Chandrasekaran, Deepa, 2010. "Extent and impact of response biases in cross-national survey research," International Journal of Research in Marketing, Elsevier, vol. 27(4), pages 329-341.
    8. Caeyers, Bet & Chalmers, Neil & De Weerdt, Joachim, 2012. "Improving consumption measurement and other survey data through CAPI: Evidence from a randomized experiment," Journal of Development Economics, Elsevier, vol. 98(1), pages 19-33.
    9. Reisinger, James, 2022. "Subjective well-being and social desirability," Journal of Public Economics, Elsevier, vol. 214(C).
    10. Feng, Shihan & Huang, Feng, 2024. "Does survey mode matter? An experimental evaluation of data quality in China," China Economic Review, Elsevier, vol. 88(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zawojska, Ewa & Czajkowski, Mikotaj, "undated". "Are preferences stated in web vs. personal interviews different? A comparison of willingness to pay results for a large multi-country study of the Baltic Sea eutrophication reduction," Annual Meeting, 2017, June 18-21, Montreal, Canada 258604, Canadian Agricultural Economics Society.
    2. Daniel Gerszon Mahler & Elizabeth Foster & Samuel Tetteh-Baah, 2024. "How Improved Household Surveys Influence National and International Poverty Rates," World Bank Publications - Reports 42225, The World Bank Group.
    3. Böhme, Marcus & Stöhr, Tobias, 2012. "Guidelines for the use of household interview duration analysis in CAPI survey management," Kiel Working Papers 1779, Kiel Institute for the World Economy.
    4. Debra Wright & Matt Sloan & Kirsten Barrett, 2012. "Is There a Trade-off Between Quality and Cost? Telephone Versus Face-to-Face Interviewing of Persons with Disabilities," Mathematica Policy Research Reports cb6067df035641e99a913d534, Mathematica Policy Research.
    5. Reitmann, Ann-Kristin & Goedhuys, Micheline & Grimm, Michael & Nillesen, Eleonora E.M., 2020. "Gender attitudes in the Arab region – The role of framing and priming effects," Journal of Economic Psychology, Elsevier, vol. 80(C).
    6. Mamine, Fateh & Fares, M'hand & Minviel, Jean Joseph, 2020. "Contract Design for Adoption of Agrienvironmental Practices: A Meta-analysis of Discrete Choice Experiments," Ecological Economics, Elsevier, vol. 176(C).
    7. Jorre T. A. Vannieuwenhuyze & Geert Loosveldt, 2013. "Evaluating Relative Mode Effects in Mixed-Mode Surveys:," Sociological Methods & Research, , vol. 42(1), pages 82-104, February.
    8. Guimarães, Maria Helena & Nunes, Luís Catela & Madureira, Lívia & Santos, José Lima & Boski, Tomasz & Dentinho, Tomaz, 2015. "Measuring birdwatchers preferences: A case for using online networks and mixed-mode surveys," Tourism Management, Elsevier, vol. 46(C), pages 102-113.
    9. Mackeben, Jan, 2020. "Mode Effects in the Fourth Wave of the Linked Personnel Panel (LPP) Employee Survey," FDZ Methodenreport 202005_en, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    10. Sarker, Prianka & Hughes, Laurie & Malik, Tegwen & Dwivedi, Yogesh K., 2025. "Examining consumer adoption of social commerce: An extended META-UTAUT model," Technological Forecasting and Social Change, Elsevier, vol. 212(C).
    11. Giorgio Piccitto & Hans M. A. Schadee & Gabriele Ballarino, 2023. "Job Satisfaction and Gender in Italy: A Structural Equation Modeling Approach," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 169(3), pages 775-793, October.
    12. Joachim De Weerdt & Kalle Hirvonen, 2016. "Risk Sharing and Internal Migration," Economic Development and Cultural Change, University of Chicago Press, vol. 65(1), pages 63-86.
    13. Pape,Utz Johann & Wollburg,Philip Randolph, 2019. "Estimation of Poverty in Somalia Using Innovative Methodologies," Policy Research Working Paper Series 8735, The World Bank.
    14. Leonardo Bursztyn & Ingar Haaland & Nicolas Röver & Christopher Roth, 2025. "The Social Desirability Atlas," ECONtribute Discussion Papers Series 365, University of Bonn and University of Cologne, Germany.
    15. Joachim De Weerdt & Garance Genicot & Alice Mesnard, 2019. "Asymmetry of Information within Family Networks," Journal of Human Resources, University of Wisconsin Press, vol. 54(1), pages 225-254.
    16. Feng, Shihan & Huang, Feng, 2024. "Does survey mode matter? An experimental evaluation of data quality in China," China Economic Review, Elsevier, vol. 88(C).
    17. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, IZA Network @ LISER.
    18. repec:lic:licosd:36515 is not listed on IDEAS
    19. repec:iab:iabfme:202005(en is not listed on IDEAS
    20. Stefania Capecchi & Romina Gambacorta & Rosaria Simone & Domenico Piccolo, 2024. "Modelling cognitive response patterns to survey questions using the class of CUB models," Questioni di Economia e Finanza (Occasional Papers) 885, Bank of Italy, Economic Research and International Relations Area.
    21. Czajkowski, Mikołaj & Zawojska, Ewa & Meade, Norman & da Motta, Ronaldo Seroa & Welsh, Mike & Ortiz, Ramon Arigoni, 2024. "On the inference about a willingness-to-pay distribution using contingent valuation data," Ecological Economics, Elsevier, vol. 222(C).
    22. Lana M. Agraib & Buthaina Alkhatib & Huda Al Hourani & Islam Al-Shami, 2023. "Are online and face-to-face questionnaires equally valid and reliable methods of assessing preconception care?," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(6), pages 5563-5576, December.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:socarx:bc7qn_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://arabixiv.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.