IDEAS home Printed from https://ideas.repec.org/a/eee/poleco/v78y2023ics0176268023000265.html
   My bibliography  Save this article

Can internet surveys represent the entire population? A practitioners’ analysis

Author

Listed:
  • Grewenig, Elisabeth
  • Lergetporer, Philipp
  • Simon, Lisa
  • Werner, Katharina
  • Woessmann, Ludger

Abstract

A general concern with the representativeness of internet surveys is that they exclude the “offline” population that does not use the internet. We run a large-scale opinion survey with (1) onliners in internet-survey mode, (2) offliners in face-to-face mode, and (3) internet users in face-to-face mode. We find marked response differences between onliners and offliners in different modes (1 vs. 2). Response differences between onliners and offliners in the same face-to-face mode (2 vs. 3) disappear when controlling for background characteristics, indicating mode effects rather than unobserved population differences. Differences in background characteristics of onliners in the two modes (1 vs. 3) indicate that mode effects partly reflect sampling differences. In our setting, re-weighting online-survey observations appears a pragmatic solution when aiming at representativeness for the entire population.

Suggested Citation

  • Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2023. "Can internet surveys represent the entire population? A practitioners’ analysis," European Journal of Political Economy, Elsevier, vol. 78(C).
    • Handle: RePEc:eee:poleco:v:78:y:2023:i:c:s0176268023000265
    DOI: 10.1016/j.ejpoleco.2023.102382
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0176268023000265
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ejpoleco.2023.102382?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Malhotra, Neil & Krosnick, Jon A., 2007. "The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples," Political Analysis, Cambridge University Press, vol. 15(3), pages 286-323, July.
    2. Olivier Armantier & Scott Nelson & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2016. "The Price Is Right: Updating Inflation Expectations in a Randomized Price Information Experiment," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 503-523, July.
    3. Jérôme Hergueux & Nicolas Jacquemet, 2015. "Social preferences in the online laboratory: a randomized experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 251-283, June.
    4. Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & West, Martin R. & Woessmann, Ludger, 2018. "How information affects support for education spending: Evidence from survey experiments in Germany and the United States," Journal of Public Economics, Elsevier, vol. 167(C), pages 138-157.
    5. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    6. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    7. Sonja Settele, 2022. "How Do Beliefs about the Gender Wage Gap Affect the Demand for Public Policy?," American Economic Journal: Economic Policy, American Economic Association, vol. 14(2), pages 475-508, May.
    8. Ingvild Almås & Alexander W. Cappelen & Bertil Tungodden, 2020. "Cutthroat Capitalism versus Cuddly Socialism: Are Americans More Meritocratic and Efficiency-Seeking than Scandinavians?," Journal of Political Economy, University of Chicago Press, vol. 128(5), pages 1753-1788.
    9. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    10. Aksoy, Cevat Giray & Cabrales, Antonio & Dolls, Mathias & Durante, Ruben & Windsteiger, Lisa, 2021. "Calamities, Common Interests, Shared Identity: What Shapes Altruism and Reciprocity?," IZA Discussion Papers 14411, Institute of Labor Economics (IZA).
    11. Freundl Vera & Grewenig Elisabeth & Kugler Franziska & Lergetporer Philipp & Schüler Ruth & Wedel Katharina & Werner Katharina & Wirth Olivia & Woessmann Ludger, 2023. "The ifo Education Survey 2014–2021: A New Dataset on Public Preferences for Education Policy in Germany," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 243(6), pages 699-710, December.
    12. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    13. Ilyana Kuziemko & Michael I. Norton & Emmanuel Saez & Stefanie Stantcheva, 2015. "How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments," American Economic Review, American Economic Association, vol. 105(4), pages 1478-1508, April.
    14. Lergetporer, Philipp & Woessmann, Ludger, 2022. "Income Contingency and the Electorate's Support for Tuition," IZA Discussion Papers 14991, Institute of Labor Economics (IZA).
    15. Alberto Alesina & Stefanie Stantcheva & Edoardo Teso, 2018. "Intergenerational Mobility and Preferences for Redistribution," American Economic Review, American Economic Association, vol. 108(2), pages 521-554, February.
    16. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    17. Chesney, Thomas & Chuah, Swee-Hoon & Hoffmann, Robert, 2009. "Virtual world experimentation: An exploratory study," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 618-635, October.
    18. Cevat Giray Aksoy & Antonio Cabrales & Mathias Dolls & Ruben Durante & Lisa Windsteiger, 2021. "Calamities, Common Interests, Shared Identity: What Shapes Social Cohesion in Europe?," CESifo Working Paper Series 9112, CESifo.
    19. Gary Solon & Steven J. Haider & Jeffrey M. Wooldridge, 2015. "What Are We Weighting For?," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 301-316.
    20. Ofra Amir & David G Rand & Ya'akov Kobi Gal, 2012. "Economic Games on the Internet: The Effect of $1 Stakes," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-4, February.
    21. Haaland, Ingar & Roth, Christopher, 2020. "Labor market concerns and support for immigration," Journal of Public Economics, Elsevier, vol. 191(C).
    22. Anderhub, Vital & Muller, Rudolf & Schmidt, Carsten, 2001. "Design and evaluation of an economic experiment via the Internet," Journal of Economic Behavior & Organization, Elsevier, vol. 46(2), pages 227-247, October.
    23. Annette Jäckle & Caroline Roberts & Peter Lynn, 2010. "Assessing the Effect of Data Collection Mode on Measurement," International Statistical Review, International Statistical Institute, vol. 78(1), pages 3-20, April.
    24. Philipp Lergetporer & Guido Schwerdt & Katharina Werner & Ludger Woessmann, 2016. "Information and Preferences for Public Spending: Evidence from Representative Survey Experiments," Working Paper Series of the Department of Economics, University of Konstanz 2016-07, Department of Economics, University of Konstanz.
    25. Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2018. "Measuring and Bounding Experimenter Demand," American Economic Review, American Economic Association, vol. 108(11), pages 3266-3302, November.
    26. Berrens, Robert P. & Bohara, Alok K. & Jenkins-Smith, Hank & Silva, Carol & Weimer, David L., 2003. "The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples," Political Analysis, Cambridge University Press, vol. 11(1), pages 1-22, January.
    27. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    28. Barton, Jared & Pan, Xiaofei, 2022. "Movin’ on up? A survey experiment on mobility enhancing policies," European Journal of Political Economy, Elsevier, vol. 74(C).
    29. Leonardo Bursztyn & Alessandra L. González & David Yanagizawa-Drott, 2020. "Misperceived Social Norms: Women Working Outside the Home in Saudi Arabia," American Economic Review, American Economic Association, vol. 110(10), pages 2997-3029, October.
    30. Abraham, Martin & Lorek, Kerstin & Richter, Friedemann & Wrede, Matthias, 2018. "Breaking the norms: When is evading inheritance taxes socially acceptable?," European Journal of Political Economy, Elsevier, vol. 52(C), pages 85-102.
    31. Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2021. "Does Ignorance of Economic Returns and Costs Explain the Educational Aspiration Gap? Representative Evidence from Adults and Adolescents," Economica, London School of Economics and Political Science, vol. 88(351), pages 624-670, July.
    32. Couper, Mick P. & Kapteyn, Arie & Schonlau, Matthias & Winter, Joachim, 2007. "Noncoverage and nonresponse in an Internet survey," Munich Reprints in Economics 20093, University of Munich, Department of Economics.
    33. repec:ces:ifosdt:v::y:2017:i:17:p:17-38 is not listed on IDEAS
    34. Dolls, Mathias & Wehrhöfer, Nils, 2021. "Attitudes towards euro area reforms: Evidence from a randomized survey experiment," European Journal of Political Economy, Elsevier, vol. 67(C).
    35. Ludger Wößmann & Philipp Lergetporer & Elisabeth Grewenig & Franziska Kugler & Katharina Werner & Franziska Pfaehler, 2017. "Fürchten sich die Deutschen vor der Digitalisierung? Ergebnisse des ifo Bildungsbarometers 2017," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 70(17), pages 17-38, September.
    36. Sonja Settele, 2022. "How Do Beliefs About the Gender Wage Gap Affect the Demand for Public Policy?," ECONtribute Discussion Papers Series 179, University of Bonn and University of Cologne, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Guizzo Altube, Matías & Scartascini, Carlos, 2024. "Gender-Based Research and Interviewer Effects: Evidence for Latin America and the Caribbean," IDB Publications (Working Papers) 13475, Inter-American Development Bank.
    2. Angerer, Silvia & Glätzle-Rützler, Daniela & Lergetporer, Philipp & Rittmannsberger, Thomas, 2024. "Beliefs about social norms and gender-based polarization of COVID-19 vaccination readiness," European Economic Review, Elsevier, vol. 163(C).
    3. Breyer, Friedrich & Breunig, Christian & Kapteina, Mark & Schwerdt, Guido & Sterba, Maj-Britt, 2024. "Between Beveridge and Bismarck: Preferences for redistribution through public pensions," Working Papers 23, University of Konstanz, Cluster of Excellence "The Politics of Inequality. Perceptions, Participation and Policies".
    4. Alessandra Casarico & Jana Schuetz & Silke Uebelmesser, 2024. "Gender Inequality over the Life Cycle, Information Provision and Policy Preferences," CESifo Working Paper Series 10916, CESifo.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, Institute of Labor Economics (IZA).
    2. Lasse J. Jessen & Sebastian Koehne & Patrick Nüß & Jens Ruhose, 2024. "Socioeconomic Inequality in Life Expectancy: Perception and Policy Demand," CESifo Working Paper Series 10940, CESifo.
    3. Lergetporer, P & Woessmann, L, 2022. "Income Contingency and the Electorates Support for Tuition," CAGE Online Working Paper Series 606, Competitive Advantage in the Global Economy (CAGE).
    4. Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2022. "Incentives, search engines, and the elicitation of subjective beliefs: Evidence from representative online survey experiments," Journal of Econometrics, Elsevier, vol. 231(1), pages 304-326.
    5. Lergetporer, Philipp & Woessmann, Ludger, 2023. "Earnings information and public preferences for university tuition: Evidence from representative experiments," Journal of Public Economics, Elsevier, vol. 226(C).
    6. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    7. Starkov, Egor, 2023. "Only time will tell: Credible dynamic signaling," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    8. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2023. "Designing Information Provision Experiments," Journal of Economic Literature, American Economic Association, vol. 61(1), pages 3-40, March.
    9. Lergetporer, Philipp & Piopiunik, Marc & Simon, Lisa, 2021. "Does the education level of refugees affect natives’ attitudes?," European Economic Review, Elsevier, vol. 134(C).
    10. Sebastian Blesse & Philipp Lergetporer & Justus Nover & Katharina Werner, 2023. "Transparency and Policy Competition: Experimental Evidence from German Citizens and Politicians," Rationality and Competition Discussion Paper Series 387, CRC TRR 190 Rationality and Competition.
    11. Aljosha Henkel & Ernst Fehr & Julien Senn & Thomas Epper, 2024. "Beliefs about Inequality and the Nature of Support for Redistribution," Working Papers 2024-iRisk-02, IESEG School of Management.
    12. Guo, Yiting & Shachat, Jason & Walker, Matthew J. & Wei, Lijia, 2023. "On the generalizability of using mobile devices to conduct economic experiments," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 106(C).
    13. Freundl Vera & Grewenig Elisabeth & Kugler Franziska & Lergetporer Philipp & Schüler Ruth & Wedel Katharina & Werner Katharina & Wirth Olivia & Woessmann Ludger, 2023. "The ifo Education Survey 2014–2021: A New Dataset on Public Preferences for Education Policy in Germany," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 243(6), pages 699-710, December.
    14. Riccardo Bruni & Alessandro Gioffré & Maria Marino, 2022. ""In-group bias in preferences for redistribution: a survey experiment in Italy"," IREA Working Papers 202223, University of Barcelona, Research Institute of Applied Economics, revised Nov 2023.
    15. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    16. McNamara, Trent & Mosquera, Roberto, 2024. "The political divide: The case of expectations and preferences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 110(C).
    17. Ingar Haaland & Christopher Roth, 2023. "Beliefs about Racial Discrimination and Support for Pro-Black Policies," The Review of Economics and Statistics, MIT Press, vol. 105(1), pages 40-53, January.
    18. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    19. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    20. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.

    More about this item

    Keywords

    Internet survey; Representativeness; Mode effects; Offliner; Public opinion;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • D91 - Microeconomics - - Micro-Based Behavioral Economics - - - Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making
    • I20 - Health, Education, and Welfare - - Education - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:poleco:v:78:y:2023:i:c:s0176268023000265. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/505544 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.