IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_7556.html
   My bibliography  Save this paper

Incentives, search engines, and the elicitation of subjective beliefs: evidence from representative online survey experiments

Author

Listed:
  • Elisabeth Grewenig
  • Philipp Lergetporer
  • Katharina Werner
  • Ludger Woessmann

Abstract

A large literature studies subjective beliefs about economic facts using unincentivized survey questions. We devise randomized experiments in a representative online survey to investigate whether incentivizing belief accuracy affects stated beliefs about average earnings by professional degree and average public school spending. Incentive provision does not impact earnings beliefs, but improves school-spending beliefs. Response patterns suggest that the latter effect likely reflects increased online-search activity. Consistently, an experiment that just encourages search-engine usage produces very similar results. Another experiment provides no evidence of experimenter-demand effects. Overall, results suggest that incentive provision does not reduce bias in our survey-based belief measures.

Suggested Citation

  • Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2019. "Incentives, search engines, and the elicitation of subjective beliefs: evidence from representative online survey experiments," CESifo Working Paper Series 7556, CESifo.
  • Handle: RePEc:ces:ceswps:_7556
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp7556.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    2. Adeline Delavande & Xavier Giné & David McKenzie, 2011. "Eliciting probabilistic expectations with visual aids in developing countries: how sensitive are answers to variations in elicitation design?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(3), pages 479-497, April.
    3. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    4. Grigorieff, Alexis & Roth, Christopher & Ubfal, Diego, 2016. "Does Information Change Attitudes Towards Immigrants? Representative Evidence from Survey Experiments," IZA Discussion Papers 10419, Institute of Labor Economics (IZA).
    5. Luis Armona & Andreas Fuster & Basit Zafar, 2019. "Home Price Expectations and Behaviour: Evidence from a Randomized Information Experiment," Review of Economic Studies, Oxford University Press, vol. 86(4), pages 1371-1410.
    6. Charles F. Manski, 2018. "Survey Measurement of Probabilistic Macroeconomic Expectations: Progress and Promise," NBER Macroeconomics Annual, University of Chicago Press, vol. 32(1), pages 411-471.
    7. Olivier Armantier & Scott Nelson & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2016. "The Price Is Right: Updating Inflation Expectations in a Randomized Price Information Experiment," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 503-523, July.
    8. Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2018. "Measuring and Bounding Experimenter Demand," American Economic Review, American Economic Association, vol. 108(11), pages 3266-3302, November.
    9. Terrance Hurley & Jason Shogren, 2005. "An Experimental Comparison of Induced and Elicited Beliefs," Journal of Risk and Uncertainty, Springer, vol. 30(2), pages 169-188, January.
    10. Alesina, Alberto & La Ferrara, Eliana, 2005. "Preferences for redistribution in the land of opportunities," Journal of Public Economics, Elsevier, vol. 89(5-6), pages 897-931, June.
    11. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    12. Olivier Coibion & Yuriy Gorodnichenko & Saten Kumar, 2018. "How Do Firms Form Their Expectations? New Survey Evidence," American Economic Review, American Economic Association, vol. 108(9), pages 2671-2713, September.
    13. Ansolabehere, Stephen & Meredith, Marc & Snowberg, Erik, 2013. "Asking About Numbers: Why and How," Political Analysis, Cambridge University Press, vol. 21(1), pages 48-69, January.
    14. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    15. Ilyana Kuziemko & Michael I. Norton & Emmanuel Saez & Stefanie Stantcheva, 2015. "How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments," American Economic Review, American Economic Association, vol. 105(4), pages 1478-1508, April.
    16. N. Gregory Mankiw & Ricardo Reis & Justin Wolfers, 2004. "Disagreement about Inflation Expectations," NBER Chapters, in: NBER Macroeconomics Annual 2003, Volume 18, pages 209-270, National Bureau of Economic Research, Inc.
    17. Armin Falk & Anke Becker & Thomas Dohmen & David Huffman & Uwe Sunde, 2016. "The Preference Survey Module: A Validated Instrument for Measuring Risk, Time, and Social Preferences," Working Papers 2016-003, Human Capital and Economic Opportunity Working Group.
    18. Bleemer, Zachary & Zafar, Basit, 2018. "Intended college attendance: Evidence from an experiment on college returns and costs," Journal of Public Economics, Elsevier, vol. 157(C), pages 184-211.
    19. Olivier Coibion & Yuriy Gorodnichenko, 2012. "What Can Survey Forecasts Tell Us about Information Rigidities?," Journal of Political Economy, University of Chicago Press, vol. 120(1), pages 116-159.
    20. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    21. Andreas Fuster & Ricardo Perez-Truglia & Mirko Wiederholt & Basit Zafar, 2022. "Expectations with Endogenous Information Acquisition: An Experimental Investigation," The Review of Economics and Statistics, MIT Press, vol. 104(5), pages 1059-1078, December.
    22. Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & West, Martin R. & Woessmann, Ludger, 2018. "How information affects support for education spending: Evidence from survey experiments in Germany and the United States," Journal of Public Economics, Elsevier, vol. 167(C), pages 138-157.
    23. Mummolo, Jonathan & Peterson, Erik, 2019. "Demand Effects in Survey Experiments: An Empirical Assessment," American Political Science Review, Cambridge University Press, vol. 113(2), pages 517-529, May.
    24. Arcidiacono, Peter & Hotz, V. Joseph & Kang, Songman, 2012. "Modeling college major choices using elicited measures of expectations and counterfactuals," Journal of Econometrics, Elsevier, vol. 166(1), pages 3-16.
    25. Ralph Stinebrickner & Todd R. Stinebrickner, 2014. "A Major in Science? Initial Beliefs and Final Outcomes for College Major and Dropout," Review of Economic Studies, Oxford University Press, vol. 81(1), pages 426-472.
    26. Katja Maria Kaufmann, 2014. "Understanding the income gradient in college attendance in Mexico: The role of heterogeneity in expected returns," Quantitative Economics, Econometric Society, vol. 5(3), pages 583-630, November.
    27. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," Rationality and Competition Discussion Paper Series 117, CRC TRR 190 Rationality and Competition.
    28. Bullock, John G. & Gerber, Alan S. & Hill, Seth J. & Huber, Gregory A., 2015. "Partisan Bias in Factual Beliefs about Politics," Quarterly Journal of Political Science, now publishers, vol. 10(4), pages 519-578, December.
    29. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    30. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    31. Orazio Attanasio & Teodora Boneva & Christopher Rauh, 2018. "Parental Beliefs about Returns to Different Types of Investments in School Children," Working Papers 2018-032, Human Capital and Economic Opportunity Working Group.
    32. Florian Zimmermann, 2020. "The Dynamics of Motivated Beliefs," American Economic Review, American Economic Association, vol. 110(2), pages 337-361, February.
    33. Mounir Karadja & Johanna Mollerstrom & David Seim, 2017. "Richer (and Holier) Than Thou? The Effect of Relative Income Improvements on Demand for Redistribution," The Review of Economics and Statistics, MIT Press, vol. 99(2), pages 201-212, May.
    34. Teodora Boneva & Christopher Rauh, 2017. "Socio-Economic Gaps in University Enrollment: The Role of Perceived Pecuniary and Non-Pecuniary Returns," Working Papers 2017-080, Human Capital and Economic Opportunity Working Group.
    35. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    36. Alberto Alesina & Stefanie Stantcheva & Edoardo Teso, 2018. "Intergenerational Mobility and Preferences for Redistribution," American Economic Review, American Economic Association, vol. 108(2), pages 521-554, February.
    37. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2018. "Educational Inequality and Public Policy Preferences: Evidence from Representative Survey Experiments," CAGE Online Working Paper Series 391, Competitive Advantage in the Global Economy (CAGE).
    38. Zachary Bleemer & Basit Zafar, 2014. "Information heterogeneity and intended college enrollment," Staff Reports 685, Federal Reserve Bank of New York.
    39. Matthew Wiswall & Basit Zafar, 2015. "Determinants of College Major Choice: Identification using an Information Experiment," Review of Economic Studies, Oxford University Press, vol. 82(2), pages 791-824.
    40. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    41. Martin G. Kocher & David Schindler & Stefan T. Trautmann & Yilong Xu, 2019. "Risk, time pressure, and selection effects," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 216-246, March.
    42. Hugo Benítez-Silva & Moshe Buchinsky & Hiu Man Chan & Sofia Cheidvasser & John Rust, 2004. "How large is the bias in self-reported disability?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 19(6), pages 649-670.
    43. Basit Zafar, 2011. "How Do College Students Form Expectations?," Journal of Labor Economics, University of Chicago Press, vol. 29(2), pages 301-348.
    44. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    45. Biroli, Pietro & Boneva, Teodora & Raja, Akash & Rauh, Christopher, 2022. "Parental beliefs about returns to child health investments," Journal of Econometrics, Elsevier, vol. 231(1), pages 33-57.
    46. Teodora Boneva & Christopher Rauh, 2018. "Parental Beliefs about Returns to Educational Investments—The Later the Better?," Journal of the European Economic Association, European Economic Association, vol. 16(6), pages 1669-1711.
    47. Clifford, Scott & Jerit, Jennifer, 2014. "Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies," Journal of Experimental Political Science, Cambridge University Press, vol. 1(2), pages 120-131, January.
    48. Chris Belfield & Teodora Boneva & Christopher Rauh & Jonathan Shaw, 2020. "What Drives Enrolment Gaps in Further Education? The Role of Beliefs in Sequential Schooling Decisions," Economica, London School of Economics and Political Science, vol. 87(346), pages 490-529, April.
    49. Olivier Armantier & Wändi Bruine de Bruin & Simon Potter & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2013. "Measuring Inflation Expectations," Annual Review of Economics, Annual Reviews, vol. 5(1), pages 273-301, May.
    50. Adeline Delavande & Basit Zafar, 2019. "University Choice: The Role of Expected Earnings, Nonpecuniary Outcomes, and Financial Constraints," Journal of Political Economy, University of Chicago Press, vol. 127(5), pages 2343-2393.
    51. Arcidiacono, Peter, 2004. "Ability sorting and the returns to college major," Journal of Econometrics, Elsevier, vol. 121(1-2), pages 343-375.
    52. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    53. Wang, Stephanie W., 2011. "Incentive effects: The case of belief elicitation from individuals in groups," Economics Letters, Elsevier, vol. 111(1), pages 30-33, April.
    54. Alberto Cavallo & Guillermo Cruces & Ricardo Perez-Truglia, 2017. "Inflation Expectations, Learning, and Supermarket Prices: Evidence from Survey Experiments," American Economic Journal: Macroeconomics, American Economic Association, vol. 9(3), pages 1-35, July.
    55. Emin Karagözoğlu & Martin G. Kocher, 2019. "Bargaining under time pressure from deadlines," Experimental Economics, Springer;Economic Science Association, vol. 22(2), pages 419-440, June.
    56. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    57. Prior, Markus & Sood, Gaurav & Khanna, Kabir, 2015. "You Cannot be Serious: The Impact of Accuracy Incentives on Partisan Bias in Reports of Economic Perceptions," Quarterly Journal of Political Science, now publishers, vol. 10(4), pages 489-518, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Katharina Dowling & Daniel Guhl & Daniel Klapper & Martin Spann & Lucas Stich & Narine Yegoryan, 2020. "Behavioral biases in marketing," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 449-477, May.
    2. Dietmar Fehr & Johanna Mollerstrom & Ricardo Perez-Truglia, 2022. "Your Place in the World: Relative Income and Global Inequality," American Economic Journal: Economic Policy, American Economic Association, vol. 14(4), pages 232-268, November.
    3. Haaland, Ingar & Roth, Christopher & Wohlfart. Johannes, 2020. "Designing Information Provision Experiments," The Warwick Economics Research Paper Series (TWERPS) 1275, University of Warwick, Department of Economics.
    4. Brad R. Taylor, 2020. "The psychological foundations of rational ignorance: biased heuristics and decision costs," Constitutional Political Economy, Springer, vol. 31(1), pages 70-88, March.
    5. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    6. Patrick Bareinz & Fabian Koenings, 2021. "Framing of Economic News and Policy Support During a Pandemic: Evidence from an Information Experiment," Jena Economic Research Papers 2021-004, Friedrich-Schiller-University Jena.
    7. Lergetporer, Philipp & Woessmann, Ludger, 2021. "Earnings Information and Public Preferences for University Tuition: Evidence from Representative Experiments," Rationality and Competition Discussion Paper Series 294, CRC TRR 190 Rationality and Competition.
    8. Egor Starkov, 2020. "Only Time Will Tell: Credible Dynamic Signaling," Papers 2007.09568, arXiv.org, revised Jan 2022.
    9. Christopher Roth & Sonja Settele & Johannes Wohlfart, 2020. "Beliefs About Public Debt and the Demand for Government Spending," CEBI working paper series 20-05, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    10. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    11. Juerg Schweri, 2021. "Predicting polytomous career choices in healthcare using probabilistic expectations data," Health Economics, John Wiley & Sons, Ltd., vol. 30(3), pages 544-563, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2020. "Designing Information Provision Experiments," CEBI working paper series 20-20, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    2. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    3. Christopher Roth & Sonja Settele & Johannes Wohlfart, 2020. "Beliefs about Public Debt and the Demand for Government Spending," CESifo Working Paper Series 8087, CESifo.
    4. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    5. Pamela Giustinelli, "undated". "Expectations in Education: Framework, Elicitation, and Evidence," Working Papers 2022-026, Human Capital and Economic Opportunity Working Group.
    6. Elisabeth Grewenig & Philipp Lergetporer & Katharina Werner, 2020. "Gender Norms and Labor-Supply Expectations: Experimental Evidence from Adolescents," CESifo Working Paper Series 8611, CESifo.
    7. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," Rationality and Competition Discussion Paper Series 145, CRC TRR 190 Rationality and Competition.
    8. Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2021. "Does Ignorance of Economic Returns and Costs Explain the Educational Aspiration Gap? Representative Evidence from Adults and Adolescents," Economica, London School of Economics and Political Science, vol. 88(351), pages 624-670, July.
    9. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," CAGE Online Working Paper Series 405, Competitive Advantage in the Global Economy (CAGE).
    10. Lergetporer, Philipp & Woessmann, Ludger, 2021. "Earnings Information and Public Preferences for University Tuition: Evidence from Representative Experiments," Rationality and Competition Discussion Paper Series 294, CRC TRR 190 Rationality and Competition.
    11. Katharina Werner, 2019. "Der Einfluss von Informationen auf die öffentliche Meinung zur Bildung - Erkenntnisse aus repräsentativen Befragungsexperimenten," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 82.
    12. Egor Starkov, 2020. "Only Time Will Tell: Credible Dynamic Signaling," Discussion Papers 20-05, University of Copenhagen. Department of Economics.
    13. Blesse, Sebastian & Heinemann, Friedrich & Krieger, Tommy, 2021. "Informationsdefizite als Hindernis rationaler Wirtschaftspolitik: Ausmass, Ursachen und Gegenstrategien. Eine Studie mit Unterstützung der Brigitte Strube Stiftung," ZEW Expertises, ZEW - Leibniz Centre for European Economic Research, number 241989.
    14. Warn N. Lekfuangfu & Reto Odermatt, 2020. "All I have to do is dream? The role of aspirations in intergenerational mobility and well-being," CReAM Discussion Paper Series 2016, Centre for Research and Analysis of Migration (CReAM), Department of Economics, University College London.
    15. Lergetporer, Philipp & Piopiunik, Marc & Simon, Lisa, 2021. "Does the education level of refugees affect natives’ attitudes?," European Economic Review, Elsevier, vol. 134(C).
    16. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    17. Philipp Lergetporer & Marc Piopiunik & Lisa Simon, 2017. "Does the Education Level of Refugees Affect Natives' Attitudes?," CESifo Working Paper Series 6832, CESifo.
    18. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    19. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    20. Christopher Roth & Sonja Settele & Johannes Wohlfart, 2022. "Risk Exposure and Acquisition of Macroeconomic Information," American Economic Review: Insights, American Economic Association, vol. 4(1), pages 34-53, March.

    More about this item

    Keywords

    beliefs; incentives; online search; survey experiment;
    All these keywords.

    JEL classification:

    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_7556. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.