IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_7222.html
   My bibliography  Save this paper

Can Online Surveys Represent the Entire Population?

Author

Listed:
  • Elisabeth Grewenig
  • Philipp Lergetporer
  • Lisa Simon
  • Katharina Werner
  • Ludger Woessmann

Abstract

A general concern with the representativeness of online surveys is that they exclude the “offline” population that does not use the internet. We run a large-scale opinion survey with (1) onliners in web mode, (2) offliners in face-to-face mode, and (3) onliners in face-to-face mode. We find marked response differences between onliners and offliners in the mixed-mode setting (1 vs. 2). Response differences between onliners and offliners in the same face-to-face mode (2 vs. 3) disappear when controlling for background characteristics, indicating mode effects rather than unobserved population differences. Differences in background characteristics of onliners in the two modes (1 vs. 3) indicate that mode effects partly reflect sampling differences. In our setting, re-weighting online-survey observations appears a pragmatic solution when aiming at representativeness for the entire population.

Suggested Citation

  • Elisabeth Grewenig & Philipp Lergetporer & Lisa Simon & Katharina Werner & Ludger Woessmann, 2018. "Can Online Surveys Represent the Entire Population?," CESifo Working Paper Series 7222, CESifo.
  • Handle: RePEc:ces:ceswps:_7222
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp7222.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jérôme Hergueux & Nicolas Jacquemet, 2015. "Social preferences in the online laboratory: a randomized experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 251-283, June.
    2. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    3. Chesney, Thomas & Chuah, Swee-Hoon & Hoffmann, Robert, 2009. "Virtual world experimentation: An exploratory study," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 618-635, October.
    4. Olivier Armantier & Scott Nelson & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2016. "The Price Is Right: Updating Inflation Expectations in a Randomized Price Information Experiment," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 503-523, July.
    5. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    6. Ilyana Kuziemko & Michael I. Norton & Emmanuel Saez & Stefanie Stantcheva, 2015. "How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments," American Economic Review, American Economic Association, vol. 105(4), pages 1478-1508, April.
    7. Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2018. "Does Ignorance of Economic Returns and Costs Explain the Educational Aspiration Gap? Evidence from Representative Survey Experiments," CESifo Working Paper Series 7000, CESifo.
    8. Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & West, Martin R. & Woessmann, Ludger, 2018. "How information affects support for education spending: Evidence from survey experiments in Germany and the United States," Journal of Public Economics, Elsevier, vol. 167(C), pages 138-157.
    9. Haaland, Ingar & Roth, Christopher, 2020. "Labor market concerns and support for immigration," Journal of Public Economics, Elsevier, vol. 191(C).
    10. Anderhub, Vital & Muller, Rudolf & Schmidt, Carsten, 2001. "Design and evaluation of an economic experiment via the Internet," Journal of Economic Behavior & Organization, Elsevier, vol. 46(2), pages 227-247, October.
    11. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    12. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    13. Alberto Alesina & Stefanie Stantcheva & Edoardo Teso, 2018. "Intergenerational Mobility and Preferences for Redistribution," American Economic Review, American Economic Association, vol. 108(2), pages 521-554, February.
    14. Annette Jäckle & Caroline Roberts & Peter Lynn, 2010. "Assessing the Effect of Data Collection Mode on Measurement," International Statistical Review, International Statistical Institute, vol. 78(1), pages 3-20, April.
    15. Couper, Mick P. & Kapteyn, Arie & Schonlau, Matthias & Winter, Joachim, 2007. "Noncoverage and nonresponse in an Internet survey," Munich Reprints in Economics 20093, University of Munich, Department of Economics.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," Rationality and Competition Discussion Paper Series 145, CRC TRR 190 Rationality and Competition.
    2. Haaland, Ingar & Roth, Christopher & Wohlfart. Johannes, 2020. "Designing Information Provision Experiments," The Warwick Economics Research Paper Series (TWERPS) 1275, University of Warwick, Department of Economics.
    3. Ferdi Botha & John P. New & Sonja C. New & David C. Ribar & Nicolás Salamanca, 2021. "Implications of COVID-19 labour market shocks for inequality in financial wellbeing," Journal of Population Economics, Springer;European Society for Population Economics, vol. 34(2), pages 655-689, April.
    4. Ludger Wößmann & Philipp Lergetporer & Elisabeth Grewenig & Sarah Kersten & Katharina Werner, 2018. "Was denken die Deutschen zu Geschlechterthemen und Gleichstellung in der Bildung? – Ergebnisse des ifo Bildungsbarometers 2018," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 71(17), pages 15-30, September.
    5. Gangadharan, Lata & Harrison, Glenn W. & Leroux, Anke D., 2019. "Are risks over multiple attributes traded off? A case study of aid," Journal of Economic Behavior & Organization, Elsevier, vol. 164(C), pages 166-198.
    6. Ludger Wößmann & Vera Freundl & Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner, 2020. "Deutsche sind für mehr Einheitlichkeit und Vergleichbarkeit im Bildungssystem – Ergebnisse des ifo Bildungsbarometers 2020," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 73(09), pages 40-48, September.
    7. Christopher Roth & Sonja Settele & Johannes Wohlfart, 2020. "Beliefs About Public Debt and the Demand for Government Spending," CEBI working paper series 20-05, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    8. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    9. Egor Starkov, 2020. "Only Time Will Tell: Credible Dynamic Signaling," Discussion Papers 20-05, University of Copenhagen. Department of Economics.
    10. Knox, Melissa A. & Oddo, Vanessa M. & Walkinshaw, Lina Pinero & Jones-Smith, Jessica, 2020. "Is the public sweet on sugary beverages? Social desirability bias and sweetened beverage taxes," Economics & Human Biology, Elsevier, vol. 38(C).
    11. Sonja Settele, 2019. "How Do Beliefs about the Gender Wage Gap Affect the Demand for Public Policy?," CEBI working paper series 19-13, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    12. Elisabeth Grewenig & Sarah Kersten & Franziska Kugler & Philipp Lergetporer & Franziska Werner & Ludger Wößmann & Katharina Werner, 2019. "Was die Deutschen über Bildungsungleichheit denken," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 72(17), August.
    13. Ludger Wößmann & Vera Freundl & Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner & Larissa Zierow, 2020. "Bildung in der Coronakrise: Wie haben die Schulkinder die Zeit der Schulschließungen verbracht, und welche Bildungsmaßnahmen befürworten die Deutschen?," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 73(09), pages 25-39, September.
    14. Prabatha, Tharindu & Karunathilake, Hirushie & Mohammadpour Shotorbani, Amin & Sadiq, Rehan & Hewage, Kasun, 2021. "Community-level decentralized energy system planning under uncertainty: A comparison of mathematical models for strategy development," Applied Energy, Elsevier, vol. 283(C).
    15. Simon Hetland & Rasmus Søndergaard Pedersen & Anders Rahbek, 2019. "Dynamic Conditional Eigenvalue GARCH," Discussion Papers 19-13, University of Copenhagen. Department of Economics.
    16. Tiziana Carpi & Airo Hino & Stefano Maria Iacus & Giuseppe Porro, 2021. "Twitter Subjective Well-Being Indicator During COVID-19 Pandemic: A Cross-Country Comparative Study," Papers 2101.07695, arXiv.org.
    17. Lergetporer, Philipp & Woessmann, Ludger, 2021. "Earnings Information and Public Preferences for University Tuition: Evidence from Representative Experiments," IZA Discussion Papers 14386, Institute of Labor Economics (IZA).
    18. Sonja Settele & Cortnie Shupe, 2020. "Lives or Livelihoods? Perceived Tradeoffs and Public Demand for Non-Pharmaceutical Interventions," CEBI working paper series 20-17, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2020. "Designing Information Provision Experiments," CEBI working paper series 20-20, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    2. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    3. Hindsley, Paul & McEvoy, David M. & Morgan, O. Ashton, 2020. "Consumer Demand for Ethical Products and the Role of Cultural Worldviews: The Case of Direct-Trade Coffee," Ecological Economics, Elsevier, vol. 177(C).
    4. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    5. Hans-Theo Normann & Till Requate & Israel Waichman, 2014. "Do short-term laboratory experiments provide valid descriptions of long-term economic interactions? A study of Cournot markets," Experimental Economics, Springer;Economic Science Association, vol. 17(3), pages 371-390, September.
    6. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    7. Philipp Lergetporer & Marc Piopiunik & Lisa Simon, 2017. "Does the Education Level of Refugees Affect Natives' Attitudes?," CESifo Working Paper Series 6832, CESifo.
    8. Darya Korlyakova, 2021. "Learning about Ethnic Discrimination from Different Information Sources," CERGE-EI Working Papers wp689, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    9. Chatruc, Marisol Rodríguez & Rozo, Sandra V., 2021. "How Does It Feel to Be Part of the Minority? Impacts of Perspective-Taking on Prosocial Behaviors," IZA Discussion Papers 14303, Institute of Labor Economics (IZA).
    10. Jérôme Hergueux & Nicolas Jacquemet, 2015. "Social preferences in the online laboratory: a randomized experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 251-283, June.
    11. Lergetporer, Philipp & Piopiunik, Marc & Simon, Lisa, 2021. "Does the education level of refugees affect natives’ attitudes?," European Economic Review, Elsevier, vol. 134(C).
    12. Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2019. "Incentives, search engines, and the elicitation of subjective beliefs: evidence from representative online survey experiments," CESifo Working Paper Series 7556, CESifo.
    13. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," Rationality and Competition Discussion Paper Series 145, CRC TRR 190 Rationality and Competition.
    14. Gagnon, Nickolas & Bosmans, Kristof & Riedl, Arno, 2020. "The Effect of Unfair Chances and Gender Discrimination on Labor Supply," IZA Discussion Papers 12912, Institute of Labor Economics (IZA).
    15. Roth, Christopher & Wohlfart, Johannes, 2018. "Experienced inequality and preferences for redistribution," Journal of Public Economics, Elsevier, vol. 167(C), pages 251-262.
    16. Shigeoka, Hitoshi & Yamada, Katsunori, 2019. "Income-comparison attitudes in the United States and the United Kingdom: Evidence from discrete-choice experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 164(C), pages 414-438.
    17. Francine D. Blau & Lawrence Kahn & Nikolai Boboshko & Matthew Comey, 2021. "Th Impact of Selection into the Labor Force on the Gender Wage Gap," CESifo Working Paper Series 9103, CESifo.
    18. Rafael Di Tella & Juan Dubra, 2016. "Meet the Oligarchs: Business Legitimacy, State Capacity and Taxation," Documentos de Trabajo/Working Papers 1607, Facultad de Ciencias Empresariales y Economia. Universidad de Montevideo..
    19. Cherry, Todd L. & James, Alexander G. & Murphy, James, 2021. "The impact of public health messaging and personal experience on the acceptance of mask wearing during the COVID-19 pandemic," Journal of Economic Behavior & Organization, Elsevier, vol. 187(C), pages 415-430.
    20. Chirvi, Malte & Schneider, Cornelius, 2020. "Preferences for wealth taxation: Design, framing and the role of partisanship," arqus Discussion Papers in Quantitative Tax Research 260, arqus - Arbeitskreis Quantitative Steuerlehre.

    More about this item

    Keywords

    online survey; representativeness; mode effects; offliner; public opinion;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • D91 - Microeconomics - - Micro-Based Behavioral Economics - - - Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making
    • I20 - Health, Education, and Welfare - - Education - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_7222. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.