IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp11799.html
   My bibliography  Save this paper

Can Online Surveys Represent the Entire Population?

Author

Listed:
  • Grewenig, Elisabeth

    (Kreditanstalt für Wiederaufbau (KfW), Germany)

  • Lergetporer, Philipp

    (Technical University of Munich)

  • Simon, Lisa

    (CESifo)

  • Werner, Katharina

    (ifo Institute, University of Munich)

  • Woessmann, Ludger

    (University of Munich)

Abstract

A general concern with the representativeness of online surveys is that they exclude the "offline" population that does not use the internet. We run a large-scale opinion survey with (1) onliners in web mode, (2) offliners in face-to-face mode, and (3) onliners in face-to-face mode. We find marked response differences between onliners and offliners in the mixed-mode setting (1 vs. 2). Response differences between onliners and offliners in the same face-to-face mode (2 vs. 3) disappear when controlling for background characteristics, indicating mode effects rather than unobserved population differences. Differences in background characteristics of onliners in the two modes (1 vs. 3) indicate that mode effects partly reflect sampling differences. In our setting, re-weighting online-survey observations appears a pragmatic solution when aiming at representativeness for the entire population.

Suggested Citation

  • Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp11799
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp11799.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jérôme Hergueux & Nicolas Jacquemet, 2015. "Social preferences in the online laboratory: a randomized experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 251-283, June.
    2. Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2018. "Does Ignorance of Economic Returns and Costs Explain the Educational Aspiration Gap? Evidence from Representative Survey Experiments," CESifo Working Paper Series 7000, CESifo.
    3. Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & West, Martin R. & Woessmann, Ludger, 2018. "How information affects support for education spending: Evidence from survey experiments in Germany and the United States," Journal of Public Economics, Elsevier, vol. 167(C), pages 138-157.
    4. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    5. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    6. Ilyana Kuziemko & Michael I. Norton & Emmanuel Saez & Stefanie Stantcheva, 2015. "How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments," American Economic Review, American Economic Association, vol. 105(4), pages 1478-1508, April.
    7. Chesney, Thomas & Chuah, Swee-Hoon & Hoffmann, Robert, 2009. "Virtual world experimentation: An exploratory study," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 618-635, October.
    8. Ofra Amir & David G Rand & Ya'akov Kobi Gal, 2012. "Economic Games on the Internet: The Effect of $1 Stakes," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-4, February.
    9. Haaland, Ingar & Roth, Christopher, 2020. "Labor market concerns and support for immigration," Journal of Public Economics, Elsevier, vol. 191(C).
    10. Anderhub, Vital & Muller, Rudolf & Schmidt, Carsten, 2001. "Design and evaluation of an economic experiment via the Internet," Journal of Economic Behavior & Organization, Elsevier, vol. 46(2), pages 227-247, October.
    11. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    12. repec:hal:pseose:halshs-00984211 is not listed on IDEAS
    13. Annette Jäckle & Caroline Roberts & Peter Lynn, 2010. "Assessing the Effect of Data Collection Mode on Measurement," International Statistical Review, International Statistical Institute, vol. 78(1), pages 3-20, April.
    14. Couper, Mick P. & Kapteyn, Arie & Schonlau, Matthias & Winter, Joachim, 2007. "Noncoverage and nonresponse in an Internet survey," Munich Reprints in Economics 20093, University of Munich, Department of Economics.
    15. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    16. Olivier Armantier & Scott Nelson & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2016. "The Price Is Right: Updating Inflation Expectations in a Randomized Price Information Experiment," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 503-523, July.
    17. Alberto Alesina & Stefanie Stantcheva & Edoardo Teso, 2018. "Intergenerational Mobility and Preferences for Redistribution," American Economic Review, American Economic Association, vol. 108(2), pages 521-554, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Riccardo Bruni & Alessandro Gioffré & Maria Marino, 2022. ""In-group bias in preferences for redistribution: a survey experiment in Italy"," IREA Working Papers 202223, University of Barcelona, Research Institute of Applied Economics, revised Nov 2023.
    2. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," Rationality and Competition Discussion Paper Series 145, CRC TRR 190 Rationality and Competition.
    3. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2023. "Designing Information Provision Experiments," Journal of Economic Literature, American Economic Association, vol. 61(1), pages 3-40, March.
    4. Ludger Wößmann & Philipp Lergetporer & Elisabeth Grewenig & Sarah Kersten & Katharina Werner, 2018. "Was denken die Deutschen zu Geschlechterthemen und Gleichstellung in der Bildung? – Ergebnisse des ifo Bildungsbarometers 2018," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 71(17), pages 15-30, September.
    5. Gangadharan, Lata & Harrison, Glenn W. & Leroux, Anke D., 2019. "Are risks over multiple attributes traded off? A case study of aid," Journal of Economic Behavior & Organization, Elsevier, vol. 164(C), pages 166-198.
    6. Ludger Wößmann & Vera Freundl & Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner, 2020. "Deutsche sind für mehr Einheitlichkeit und Vergleichbarkeit im Bildungssystem – Ergebnisse des ifo Bildungsbarometers 2020," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 73(09), pages 40-48, September.
    7. Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2022. "Incentives, search engines, and the elicitation of subjective beliefs: Evidence from representative online survey experiments," Journal of Econometrics, Elsevier, vol. 231(1), pages 304-326.
    8. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    9. Starkov, Egor, 2023. "Only time will tell: Credible dynamic signaling," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    10. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    11. Cattaneo, Cristina & Grieco, Daniela, 2021. "Turning opposition into support to immigration: The role of narratives," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 785-801.
    12. Lergetporer, P & Woessmann, L, 2022. "Income Contingency and the Electorates Support for Tuition," CAGE Online Working Paper Series 606, Competitive Advantage in the Global Economy (CAGE).
    13. Sonja Settele, 2019. "How Do Beliefs about the Gender Wage Gap Affect the Demand for Public Policy?," CEBI working paper series 19-13, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    14. Angerer, Silvia & Glätzle-Rützler, Daniela & Lergetporer, Philipp & Rittmannsberger, Thomas, 2023. "How does the vaccine approval procedure affect COVID-19 vaccination intentions?," European Economic Review, Elsevier, vol. 158(C).
    15. Prabatha, Tharindu & Karunathilake, Hirushie & Mohammadpour Shotorbani, Amin & Sadiq, Rehan & Hewage, Kasun, 2021. "Community-level decentralized energy system planning under uncertainty: A comparison of mathematical models for strategy development," Applied Energy, Elsevier, vol. 283(C).
    16. Simon Hetland & Rasmus Søndergaard Pedersen & Anders Rahbek, 2019. "Dynamic Conditional Eigenvalue GARCH," Discussion Papers 19-13, University of Copenhagen. Department of Economics.
    17. Tiziana Carpi & Airo Hino & Stefano Maria Iacus & Giuseppe Porro, 2021. "Twitter Subjective Well-Being Indicator During COVID-19 Pandemic: A Cross-Country Comparative Study," Papers 2101.07695, arXiv.org.
    18. Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner, 2020. "Gender Norms and Labor-Supply Expectations: Experimental Evidence from Adolescents," CESifo Working Paper Series 8611, CESifo.
    19. Sonja Settele, 2022. "How Do Beliefs About the Gender Wage Gap Affect the Demand for Public Policy?," ECONtribute Discussion Papers Series 179, University of Bonn and University of Cologne, Germany.
    20. Ferdi Botha & John P. New & Sonja C. New & David C. Ribar & Nicolás Salamanca, 2021. "Implications of COVID-19 labour market shocks for inequality in financial wellbeing," Journal of Population Economics, Springer;European Society for Population Economics, vol. 34(2), pages 655-689, April.
    21. Knox, Melissa A. & Oddo, Vanessa M. & Walkinshaw, Lina Pinero & Jones-Smith, Jessica, 2020. "Is the public sweet on sugary beverages? Social desirability bias and sweetened beverage taxes," Economics & Human Biology, Elsevier, vol. 38(C).
    22. Elisabeth Grewenig & Sarah Kersten & Franziska Kugler & Philipp Lergetporer & Franziska Werner & Ludger Wößmann & Katharina Werner, 2019. "Was die Deutschen über Bildungsungleichheit denken," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 72(17), August.
    23. Lergetporer, Philipp & Woessmann, Ludger, 2023. "Earnings information and public preferences for university tuition: Evidence from representative experiments," Journal of Public Economics, Elsevier, vol. 226(C).
    24. Ludger Wößmann & Vera Freundl & Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner & Larissa Zierow, 2020. "Bildung in der Coronakrise: Wie haben die Schulkinder die Zeit der Schulschließungen verbracht, und welche Bildungsmaßnahmen befürworten die Deutschen?," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 73(09), pages 25-39, September.
    25. Sonja Settele & Cortnie Shupe, 2020. "Lives or Livelihoods? Perceived Tradeoffs and Public Demand for Non-Pharmaceutical Interventions," CEBI working paper series 20-17, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2023. "Can internet surveys represent the entire population? A practitioners’ analysis," European Journal of Political Economy, Elsevier, vol. 78(C).
    2. Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2022. "Incentives, search engines, and the elicitation of subjective beliefs: Evidence from representative online survey experiments," Journal of Econometrics, Elsevier, vol. 231(1), pages 304-326.
    3. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2023. "Designing Information Provision Experiments," Journal of Economic Literature, American Economic Association, vol. 61(1), pages 3-40, March.
    4. Guo, Yiting & Shachat, Jason & Walker, Matthew J. & Wei, Lijia, 2023. "On the generalizability of using mobile devices to conduct economic experiments," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 106(C).
    5. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    6. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    7. Blesse, Sebastian & Heinemann, Friedrich & Krieger, Tommy, 2021. "Informationsdefizite als Hindernis rationaler Wirtschaftspolitik: Ausmass, Ursachen und Gegenstrategien. Eine Studie mit Unterstützung der Brigitte Strube Stiftung," ZEW Expertises, ZEW - Leibniz Centre for European Economic Research, number 241989.
    8. Hyndman, Kyle & Walker, Matthew J., 2022. "Fairness and risk in ultimatum bargaining," Games and Economic Behavior, Elsevier, vol. 132(C), pages 90-105.
    9. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    10. Lasse J. Jessen & Sebastian Koehne & Patrick Nüß & Jens Ruhose, 2024. "Socioeconomic Inequality in Life Expectancy: Perception and Policy Demand," CESifo Working Paper Series 10940, CESifo.
    11. Lergetporer, Philipp & Piopiunik, Marc & Simon, Lisa, 2021. "Does the education level of refugees affect natives’ attitudes?," European Economic Review, Elsevier, vol. 134(C).
    12. Jérôme Hergueux & Nicolas Jacquemet, 2015. "Social preferences in the online laboratory: a randomized experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 251-283, June.
    13. Lergetporer, Philipp & Woessmann, Ludger, 2023. "Earnings information and public preferences for university tuition: Evidence from representative experiments," Journal of Public Economics, Elsevier, vol. 226(C).
    14. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    15. Hindsley, Paul & McEvoy, David M. & Morgan, O. Ashton, 2020. "Consumer Demand for Ethical Products and the Role of Cultural Worldviews: The Case of Direct-Trade Coffee," Ecological Economics, Elsevier, vol. 177(C).
    16. Adams-Prassl, Abi & Boneva, Teodora & Golin, Marta & Rauh, Christopher, 2023. "The value of sick pay," European Economic Review, Elsevier, vol. 151(C).
    17. Lergetporer, Philipp & Woessmann, Ludger, 2022. "Income Contingency and the Electorate's Support for Tuition," IZA Discussion Papers 14991, Institute of Labor Economics (IZA).
    18. Ingar Haaland & Christopher Roth, 2023. "Beliefs about Racial Discrimination and Support for Pro-Black Policies," The Review of Economics and Statistics, MIT Press, vol. 105(1), pages 40-53, January.
    19. Starkov, Egor, 2023. "Only time will tell: Credible dynamic signaling," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    20. Shigeoka, Hitoshi & Yamada, Katsunori, 2019. "Income-comparison attitudes in the United States and the United Kingdom: Evidence from discrete-choice experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 164(C), pages 414-438.

    More about this item

    Keywords

    offliner; mode effects; representativeness; online survey; public opinion;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • D91 - Microeconomics - - Micro-Based Behavioral Economics - - - Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making
    • I20 - Health, Education, and Welfare - - Education - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp11799. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.