IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_7556.html
   My bibliography  Save this paper

Incentives, search engines, and the elicitation of subjective beliefs: evidence from representative online survey experiments

Author

Listed:
  • Elisabeth Grewenig
  • Philipp Lergetporer
  • Katharina Werner
  • Ludger Woessmann

Abstract

A large literature studies subjective beliefs about economic facts using unincentivized survey questions. We devise randomized experiments in a representative online survey to investigate whether incentivizing belief accuracy affects stated beliefs about average earnings by professional degree and average public school spending. Incentive provision does not impact earnings beliefs, but improves school-spending beliefs. Response patterns suggest that the latter effect likely reflects increased online-search activity. Consistently, an experiment that just encourages search-engine usage produces very similar results. Another experiment provides no evidence of experimenter-demand effects. Overall, results suggest that incentive provision does not reduce bias in our survey-based belief measures.

Suggested Citation

  • Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2019. "Incentives, search engines, and the elicitation of subjective beliefs: evidence from representative online survey experiments," CESifo Working Paper Series 7556, CESifo.
    • Handle: RePEc:ces:ceswps:_7556
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp7556.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Olivier Armantier & Scott Nelson & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2016. "The Price Is Right: Updating Inflation Expectations in a Randomized Price Information Experiment," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 503-523, July.
    2. Terrance Hurley & Jason Shogren, 2005. "An Experimental Comparison of Induced and Elicited Beliefs," Journal of Risk and Uncertainty, Springer, vol. 30(2), pages 169-188, January.
    3. Olivier Coibion & Yuriy Gorodnichenko & Saten Kumar, 2018. "How Do Firms Form Their Expectations? New Survey Evidence," American Economic Review, American Economic Association, vol. 108(9), pages 2671-2713, September.
    4. Ansolabehere, Stephen & Meredith, Marc & Snowberg, Erik, 2013. "Asking About Numbers: Why and How," Political Analysis, Cambridge University Press, vol. 21(1), pages 48-69, January.
    5. Bleemer, Zachary & Zafar, Basit, 2018. "Intended college attendance: Evidence from an experiment on college returns and costs," Journal of Public Economics, Elsevier, vol. 157(C), pages 184-211.
    6. Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & West, Martin R. & Woessmann, Ludger, 2018. "How information affects support for education spending: Evidence from survey experiments in Germany and the United States," Journal of Public Economics, Elsevier, vol. 167(C), pages 138-157.
    7. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    8. Luis Armona & Andreas Fuster & Basit Zafar, 2019. "Home Price Expectations and Behaviour: Evidence from a Randomized Information Experiment," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 86(4), pages 1371-1410.
    9. Arcidiacono, Peter & Hotz, V. Joseph & Kang, Songman, 2012. "Modeling college major choices using elicited measures of expectations and counterfactuals," Journal of Econometrics, Elsevier, vol. 166(1), pages 3-16.
    10. Grigorieff, Alexis & Roth, Christopher & Ubfal, Diego, 2016. "Does Information Change Attitudes Towards Immigrants? Representative Evidence from Survey Experiments," IZA Discussion Papers 10419, Institute of Labor Economics (IZA).
    11. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    12. Katja Maria Kaufmann, 2014. "Understanding the income gradient in college attendance in Mexico: The role of heterogeneity in expected returns," Quantitative Economics, Econometric Society, vol. 5(3), pages 583-630, November.
    13. Mounir Karadja & Johanna Mollerstrom & David Seim, 2017. "Richer (and Holier) Than Thou? The Effect of Relative Income Improvements on Demand for Redistribution," The Review of Economics and Statistics, MIT Press, vol. 99(2), pages 201-212, May.
    14. Teodora Boneva & Christopher Rauh, 2017. "Socio-Economic Gaps in University Enrollment: The Role of Perceived Pecuniary and Non-Pecuniary Returns," Working Papers 2017-080, Human Capital and Economic Opportunity Working Group.
    15. N. Gregory Mankiw & Ricardo Reis & Justin Wolfers, 2004. "Disagreement about Inflation Expectations," NBER Chapters, in: NBER Macroeconomics Annual 2003, Volume 18, pages 209-270, National Bureau of Economic Research, Inc.
    16. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    17. Zachary Bleemer & Basit Zafar, 2014. "Information heterogeneity and intended college enrollment," Staff Reports 685, Federal Reserve Bank of New York.
    18. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    19. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    20. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    21. Biroli, Pietro & Boneva, Teodora & Raja, Akash & Rauh, Christopher, 2022. "Parental beliefs about returns to child health investments," Journal of Econometrics, Elsevier, vol. 231(1), pages 33-57.
    22. Ilyana Kuziemko & Michael I. Norton & Emmanuel Saez & Stefanie Stantcheva, 2015. "How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments," American Economic Review, American Economic Association, vol. 105(4), pages 1478-1508, April.
    23. Arcidiacono, Peter, 2004. "Ability sorting and the returns to college major," Journal of Econometrics, Elsevier, vol. 121(1-2), pages 343-375.
    24. Alberto Alesina & Stefanie Stantcheva & Edoardo Teso, 2018. "Intergenerational Mobility and Preferences for Redistribution," American Economic Review, American Economic Association, vol. 108(2), pages 521-554, February.
    25. Olivier Coibion & Yuriy Gorodnichenko, 2012. "What Can Survey Forecasts Tell Us about Information Rigidities?," Journal of Political Economy, University of Chicago Press, vol. 120(1), pages 116-159.
    26. Alberto Cavallo & Guillermo Cruces & Ricardo Perez-Truglia, 2017. "Inflation Expectations, Learning, and Supermarket Prices: Evidence from Survey Experiments," American Economic Journal: Macroeconomics, American Economic Association, vol. 9(3), pages 1-35, July.
    27. Armin Falk & Anke Becker & Thomas Dohmen & David Huffman & Uwe Sunde, 2023. "The Preference Survey Module: A Validated Instrument for Measuring Risk, Time, and Social Preferences," Management Science, INFORMS, vol. 69(4), pages 1935-1950, April.
    28. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, Institute of Labor Economics (IZA).
    29. Prior, Markus & Sood, Gaurav & Khanna, Kabir, 2015. "You Cannot be Serious: The Impact of Accuracy Incentives on Partisan Bias in Reports of Economic Perceptions," Quarterly Journal of Political Science, now publishers, vol. 10(4), pages 489-518, December.
    30. Adeline Delavande & Xavier Giné & David McKenzie, 2011. "Eliciting probabilistic expectations with visual aids in developing countries: how sensitive are answers to variations in elicitation design?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(3), pages 479-497, April.
    31. Grigorieff, Alexis & Roth, Christopher & Ubfal, Diego, 2016. "Does Information Change Attitudes Towards Immigrants? Representative Evidence from Survey Experiments," IZA Discussion Papers 10419, Institute of Labor Economics (IZA).
    32. Charles F. Manski, 2018. "Survey Measurement of Probabilistic Macroeconomic Expectations: Progress and Promise," NBER Macroeconomics Annual, University of Chicago Press, vol. 32(1), pages 411-471.
    33. Andreas Fuster & Ricardo Perez-Truglia & Mirko Wiederholt & Basit Zafar, 2022. "Expectations with Endogenous Information Acquisition: An Experimental Investigation," The Review of Economics and Statistics, MIT Press, vol. 104(5), pages 1059-1078, December.
    34. Bullock, John G. & Gerber, Alan S. & Hill, Seth J. & Huber, Gregory A., 2015. "Partisan Bias in Factual Beliefs about Politics," Quarterly Journal of Political Science, now publishers, vol. 10(4), pages 519-578, December.
    35. Orazio Attanasio & Teodora Boneva & Christopher Rauh, 2022. "Parental Beliefs about Returns to Different Types of Investments in School Children," Journal of Human Resources, University of Wisconsin Press, vol. 57(6), pages 1789-1825.
    36. Matthew Wiswall & Basit Zafar, 2015. "Determinants of College Major Choice: Identification using an Information Experiment," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(2), pages 791-824.
    37. Martin G. Kocher & David Schindler & Stefan T. Trautmann & Yilong Xu, 2019. "Risk, time pressure, and selection effects," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 216-246, March.
    38. Alesina, Alberto & La Ferrara, Eliana, 2005. "Preferences for redistribution in the land of opportunities," Journal of Public Economics, Elsevier, vol. 89(5-6), pages 897-931, June.
    39. Basit Zafar, 2011. "How Do College Students Form Expectations?," Journal of Labor Economics, University of Chicago Press, vol. 29(2), pages 301-348.
    40. Ralph Stinebrickner & Todd R. Stinebrickner, 2014. "A Major in Science? Initial Beliefs and Final Outcomes for College Major and Dropout," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(1), pages 426-472.
    41. Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2018. "Measuring and Bounding Experimenter Demand," American Economic Review, American Economic Association, vol. 108(11), pages 3266-3302, November.
    42. Adeline Delavande & Basit Zafar, 2019. "University Choice: The Role of Expected Earnings, Nonpecuniary Outcomes, and Financial Constraints," Journal of Political Economy, University of Chicago Press, vol. 127(5), pages 2343-2393.
    43. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    44. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    45. Emin Karagözoğlu & Martin G. Kocher, 2019. "Bargaining under time pressure from deadlines," Experimental Economics, Springer;Economic Science Association, vol. 22(2), pages 419-440, June.
    46. Hugo Benítez-Silva & Moshe Buchinsky & Hiu Man Chan & Sofia Cheidvasser & John Rust, 2004. "How large is the bias in self-reported disability?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 19(6), pages 649-670.
    47. Teodora Boneva & Christopher Rauh, 2018. "Parental Beliefs about Returns to Educational Investments—The Later the Better?," Journal of the European Economic Association, European Economic Association, vol. 16(6), pages 1669-1711.
    48. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    49. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    50. Mummolo, Jonathan & Peterson, Erik, 2019. "Demand Effects in Survey Experiments: An Empirical Assessment," American Political Science Review, Cambridge University Press, vol. 113(2), pages 517-529, May.
    51. Florian Zimmermann, 2020. "The Dynamics of Motivated Beliefs," American Economic Review, American Economic Association, vol. 110(2), pages 337-361, February.
    52. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    53. Olivier Armantier & Wändi Bruine de Bruin & Simon Potter & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2013. "Measuring Inflation Expectations," Annual Review of Economics, Annual Reviews, vol. 5(1), pages 273-301, May.
    54. Wang, Stephanie W., 2011. "Incentive effects: The case of belief elicitation from individuals in groups," Economics Letters, Elsevier, vol. 111(1), pages 30-33, April.
    55. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    56. Clifford, Scott & Jerit, Jennifer, 2014. "Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies," Journal of Experimental Political Science, Cambridge University Press, vol. 1(2), pages 120-131, January.
    57. Chris Belfield & Teodora Boneva & Christopher Rauh & Jonathan Shaw, 2020. "What Drives Enrolment Gaps in Further Education? The Role of Beliefs in Sequential Schooling Decisions," Economica, London School of Economics and Political Science, vol. 87(346), pages 490-529, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Katharina Dowling & Daniel Guhl & Daniel Klapper & Martin Spann & Lucas Stich & Narine Yegoryan, 2020. "Behavioral biases in marketing," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 449-477, May.
    2. Dietmar Fehr & Johanna Mollerstrom & Ricardo Perez-Truglia, 2022. "Your Place in the World: Relative Income and Global Inequality," American Economic Journal: Economic Policy, American Economic Association, vol. 14(4), pages 232-268, November.
    3. Freundl Vera & Grewenig Elisabeth & Kugler Franziska & Lergetporer Philipp & Schüler Ruth & Wedel Katharina & Werner Katharina & Wirth Olivia & Woessmann Ludger, 2023. "The ifo Education Survey 2014–2021: A New Dataset on Public Preferences for Education Policy in Germany," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 243(6), pages 699-710, December.
    4. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2023. "Designing Information Provision Experiments," Journal of Economic Literature, American Economic Association, vol. 61(1), pages 3-40, March.
    5. Patrick Bareinz & Fabian Koenings, 2021. "Framing of Economic News and Policy Support During a Pandemic: Evidence from an Information Experiment," Jena Economics Research Papers 2021-004, Friedrich-Schiller-University Jena.
    6. Chantal Pezold & Simon Jäger & Patrick Nüss, 2023. "Labor Market Tightness and Union Activity," NBER Working Papers 31988, National Bureau of Economic Research, Inc.
    7. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    8. Starkov, Egor, 2023. "Only time will tell: Credible dynamic signaling," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    9. Juerg Schweri, 2021. "Predicting polytomous career choices in healthcare using probabilistic expectations data," Health Economics, John Wiley & Sons, Ltd., vol. 30(3), pages 544-563, March.
    10. Lergetporer, Philipp & Woessmann, Ludger, 2023. "Earnings information and public preferences for university tuition: Evidence from representative experiments," Journal of Public Economics, Elsevier, vol. 226(C).
    11. Angerer, Silvia & Glätzle-Rützler, Daniela & Lergetporer, Philipp & Rittmannsberger, Thomas, 2024. "Beliefs about social norms and gender-based polarization of COVID-19 vaccination readiness," European Economic Review, Elsevier, vol. 163(C).
    12. Brad R. Taylor, 2020. "The psychological foundations of rational ignorance: biased heuristics and decision costs," Constitutional Political Economy, Springer, vol. 31(1), pages 70-88, March.
    13. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2023. "Designing Information Provision Experiments," Journal of Economic Literature, American Economic Association, vol. 61(1), pages 3-40, March.
    2. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    3. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    4. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," Rationality and Competition Discussion Paper Series 145, CRC TRR 190 Rationality and Competition.
    5. Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina, 2020. "Gender Norms and Labor-Supply Expectations: Experimental Evidence from Adolescents," Rationality and Competition Discussion Paper Series 259, CRC TRR 190 Rationality and Competition.
    6. Katharina Werner, 2019. "The Role of Information for Public Preferences on Education – Evidence from Representative Survey Experiments," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 82.
    7. Pamela Giustinelli, 2022. "Expectations in Education: Framework, Elicitation, and Evidence," Working Papers 2022-026, Human Capital and Economic Opportunity Working Group.
    8. Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2021. "Does Ignorance of Economic Returns and Costs Explain the Educational Aspiration Gap? Representative Evidence from Adults and Adolescents," Economica, London School of Economics and Political Science, vol. 88(351), pages 624-670, July.
    9. Lergetporer, Philipp & Piopiunik, Marc & Simon, Lisa, 2021. "Does the education level of refugees affect natives’ attitudes?," European Economic Review, Elsevier, vol. 134(C).
    10. Lergetporer, Philipp & Woessmann, Ludger, 2023. "Earnings information and public preferences for university tuition: Evidence from representative experiments," Journal of Public Economics, Elsevier, vol. 226(C).
    11. Starkov, Egor, 2023. "Only time will tell: Credible dynamic signaling," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    12. Blesse, Sebastian & Heinemann, Friedrich & Krieger, Tommy, 2021. "Informationsdefizite als Hindernis rationaler Wirtschaftspolitik: Ausmass, Ursachen und Gegenstrategien. Eine Studie mit Unterstützung der Brigitte Strube Stiftung," ZEW Expertises, ZEW - Leibniz Centre for European Economic Research, number 241989.
    13. Lekfuangfu, Warn N. & Odermatt, Reto, 2022. "All I have to do is dream? The role of aspirations in intergenerational mobility and well-being," European Economic Review, Elsevier, vol. 148(C).
    14. Marcelo Gantier & Rafael Novella & Andrea Repetto, 2024. "Subjective expectations and schooling choices in Latin America and the Caribbean," Journal of International Development, John Wiley & Sons, Ltd., vol. 36(3), pages 1593-1621, April.
    15. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    16. Dylong, Patrick & Koenings, Fabian, 2023. "Framing of economic news and policy support during a pandemic: Evidence from a survey experiment," European Journal of Political Economy, Elsevier, vol. 76(C).
    17. Michele Giannola, 2024. "Parental Investments and Intra-household Inequality in Child Human Capital: Evidence from a Survey Experiment," The Economic Journal, Royal Economic Society, vol. 134(658), pages 671-727.
    18. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    19. Ingar Haaland & Christopher Roth, 2023. "Beliefs about Racial Discrimination and Support for Pro-Black Policies," The Review of Economics and Statistics, MIT Press, vol. 105(1), pages 40-53, January.
    20. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).

    More about this item

    Keywords

    beliefs; incentives; online search; survey experiment;
    All these keywords.

    JEL classification:

    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_7556. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.