IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/9zvqj.html
   My bibliography  Save this paper

Taking A Closer Look At The Bayesian Truth Serum: A Registered Report (Stage 2 Registered Report)

Author

Listed:
  • Schönegger, Philipp

    (University of St Andrews)

  • Verheyen, Steven

    (Erasmus University Rotterdam)

Abstract

Over the past decade, psychology and its cognate disciplines have undergone substantial scientific reform, ranging from advances in statistical methodology to significant changes in academic norms. One aspect of experimental design that has received comparatively little attention is incentivisation, i.e. the way that participants are rewarded and incentivised monetarily for their participation in experiments and surveys. While incentive-compatible designs are the norm in disciplines like economics, the majority of studies in psychology and experimental philosophy are constructed such that individuals’ incentives to maximise their payoffs in many cases stand opposed to their incentives to state their true preferences honestly. This is in part because the subject matter is often self-report data about subjective topics and the sample is drawn from online platforms like Prolific or MTurk where many participants are out to make a quick buck. One mechanism that allows for the introduction of an incentive-compatible design in such circumstances is the Bayesian Truth Serum (BTS; Prelec, 2004), which rewards participants based on how surprisingly common their answers are. Recently, Schoenegger (2021) applied this mechanism in the context of Likert-scale self-reports, finding that the introduction of this mechanism significantly altered response behaviour. In this registered report, we further investigate this mechanism by (i) attempting to directly replicate the previous result and (ii) analysing if the Bayesian Truth Serum’s effect is distinct from the effects of its constituent parts (increase in expected earnings and addition of prediction tasks). We fail to replicate the effect of the BTS on response behaviour and are as such unable to recommend wide adoption of the BTS on the basis of these data. Further, we provide weak evidence that the prediction task itself influences response distributions and that this task’s effect is distinct from an increase in expected earnings, suggesting that the BTS’s effects may ameliorate distinct effects of its constituent parts.

Suggested Citation

  • Schönegger, Philipp & Verheyen, Steven, 2022. "Taking A Closer Look At The Bayesian Truth Serum: A Registered Report (Stage 2 Registered Report)," OSF Preprints 9zvqj, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:9zvqj
    DOI: 10.31219/osf.io/9zvqj
    as

    Download full text from publisher

    File URL: https://osf.io/download/62a48834b47e7908ae6396e2/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/9zvqj?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. M. Galesic & W. Bruine de Bruin & M. Dumas & A. Kapteyn & J. E. Darling & E. Meijer, 2018. "Asking about social circles improves election predictions," Nature Human Behaviour, Nature, vol. 2(3), pages 187-193, March.
    2. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    3. Morgan R Frank & Manuel Cebrian & Galen Pickard & Iyad Rahwan, 2017. "Validating Bayesian truth serum in large-scale online human experiments," PLOS ONE, Public Library of Science, vol. 12(5), pages 1-13, May.
    4. Matthew J C Crump & John V McDonnell & Todd M Gureckis, 2013. "Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research," PLOS ONE, Public Library of Science, vol. 8(3), pages 1-18, March.
    5. Zhou, Fan & Page, Lionel & Perrons, Robert K. & Zheng, Zuduo & Washington, Simon, 2019. "Long-term forecasts for energy commodities price: What the experts think," Energy Economics, Elsevier, vol. 84(C).
    6. Paas, Leonard J. & Dolnicar, Sara & Karlsson, Logi, 2018. "Instructional Manipulation Checks: A longitudinal analysis with implications for MTurk," International Journal of Research in Marketing, Elsevier, vol. 35(2), pages 258-269.
    7. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schönegger, Philipp & Verheyen, Steven, 2022. "Taking A Closer Look At The Bayesian Truth Serum: A Registered Report (Stage 2 Registered Report)," OSF Preprints 9zvqj_v1, Center for Open Science.
    2. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    3. James C. Cox & Vjollca Sadiraj, 2018. "Incentives," Experimental Economics Center Working Paper Series 2018-01, Experimental Economics Center, Andrew Young School of Policy Studies, Georgia State University.
    4. Lee, Natalie, 2023. "Feigning ignorance for long-term gains," Games and Economic Behavior, Elsevier, vol. 138(C), pages 42-71.
    5. Lata Gangadharan & Tarun Jain & Pushkar Maitra & Joe Vecci, 2022. "Lab-in-the-field experiments: perspectives from research on gender," The Japanese Economic Review, Springer, vol. 73(1), pages 31-59, January.
    6. Clémentine Bouleau & Nicolas Jacquemet & Maël Lebreton, 2025. "How large is "large enough" ? Large-scale experimental investigation of the reliability of confidence measures," PSE Working Papers halshs-04893009, HAL.
    7. Greiner, Ben & Grünwald, Philipp & Lindner, Thomas & Lintner, Georg & Wiernsperger, Martin, 2024. "Incentives, Framing, and Reliance on Algorithmic Advice: An Experimental Study," Department for Strategy and Innovation Working Paper Series 01/2024, WU Vienna University of Economics and Business.
    8. Novella, Rafael & Ramirez, Ericka G. Rascón, 2024. "Question-order effects on judgements under uncertainty," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 109(C).
    9. Howard Kunreuther & Erwann Michel-Kerjan, 2015. "Demand for fixed-price multi-year contracts: Experimental evidence from insurance decisions," Journal of Risk and Uncertainty, Springer, vol. 51(2), pages 171-194, October.
    10. Ranganathan, Kavitha & Lejarraga, Tomás, 2021. "Elicitation of risk preferences through satisficing," Journal of Behavioral and Experimental Finance, Elsevier, vol. 32(C).
    11. Simon Gaechter & Chris Starmer & Fabio Tufano, 2022. "Measuring “group cohesion” to reveal the power of social relationships in team production," Discussion Papers 2022-12, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    12. Aycinena, Diego & Bogliacino, Francesco & Kimbrough, Erik O., 2024. "Measuring norms: Assessing the threat of social desirability bias to the Bicchieri and Xiao elicitation method," Journal of Economic Behavior & Organization, Elsevier, vol. 222(C), pages 225-239.
    13. Kimbrough, E.O. & Vostroknutov, A., 2012. "Rules, rule-following and cooperation," Research Memorandum 053, Maastricht University, Maastricht Research School of Economics of Technology and Organization (METEOR).
    14. Simon Gächter & Lingbo Huang & Martin Sefton, 2016. "Combining “real effort” with induced effort costs: the ball-catching task," Experimental Economics, Springer;Economic Science Association, vol. 19(4), pages 687-712, December.
    15. Festré, Agnès, 2018. "Do people stand by their commitments? Evidence from a classroom experiment," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 76(C), pages 1-6.
    16. Chetan Dave & Catherine Eckel & Cathleen Johnson & Christian Rojas, 2010. "Eliciting risk preferences: When is simple better?," Journal of Risk and Uncertainty, Springer, vol. 41(3), pages 219-243, December.
    17. Helena Szrek & Li-Wei Chao & Shandir Ramlagan & Karl Peltzer, 2012. "Predicting (un)healthy behavior: A comparison of risk-taking propensity measures," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 7(6), pages 716-727, November.
    18. Klein Teeselink, Bouke & Potter van Loon, Rogier J.D. & van den Assem, Martijn J. & van Dolder, Dennie, 2020. "Incentives, performance and choking in darts," Journal of Economic Behavior & Organization, Elsevier, vol. 169(C), pages 38-52.
    19. Steven Kachelmeier & Kristy Towry, 2005. "The Limitations of Experimental Design: A Case Study Involving Monetary Incentive Effects in Laboratory Markets," Experimental Economics, Springer;Economic Science Association, vol. 8(1), pages 21-33, April.
    20. Idris Idris & Saefudin Zuhri & Mohammad Efendi, 2018. "Determinants of Employee Work Satisfaction," International Review of Management and Marketing, Econjournals, vol. 8(6), pages 47-52.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:9zvqj. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.