IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_6516.html
   My bibliography  Save this paper

Measuring and Bounding Experimenter Demand

Author

Listed:
  • Jonathan de Quidt
  • Johannes Haushofer
  • Christopher Roth

Abstract

We propose a technique for assessing robustness of behavioral measures and treatment effects to experimenter demand effects. The premise is that by deliberately inducing demand in a structured way we can measure its influence and construct plausible bounds on demand-free behavior. We provide formal restrictions on choice that validate our method, and a Bayesian model that microfounds them. Seven pre-registered experiments with eleven canonical laboratory games and around 19,000 participants demonstrate the technique. We also illustrate how demand sensitivity varies by task, participant pool, gender, real versus hypothetical incentives, and participant attentiveness, and provide both reduced-form and structural analyses of demand effects.

Suggested Citation

  • Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2017. "Measuring and Bounding Experimenter Demand," CESifo Working Paper Series 6516, CESifo.
    • Handle: RePEc:ces:ceswps:_6516
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp6516.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Barmettler, Franziska & Fehr, Ernst & Zehnder, Christian, 2012. "Big experimenter is watching you! Anonymity and prosocial behavior in the laboratory," Games and Economic Behavior, Elsevier, vol. 75(1), pages 17-34.
    2. Aurelio Bruzzo, 2018. "Recenti iniziative europee ed italiane per la valorizzazione del patrimonio culturale," Working Papers 2018127, University of Ferrara, Department of Economics.
    3. Stefano DellaVigna & Devin Pope, 2018. "What Motivates Effort? Evidence and Expert Forecasts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(2), pages 1029-1069.
    4. Stefano DellaVigna & John A. List & Ulrike Malmendier, 2012. "Testing for Altruism and Social Pressure in Charitable Giving," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(1), pages 1-56.
    5. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 631-645.
    6. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 151-184, February.
    7. Urs Fischbacher & Franziska Föllmi-Heusi, 2013. "Lies In Disguise—An Experimental Study On Cheating," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 525-547, June.
    8. Ingvild Almås & Alexander W. Cappelen & Bertil Tungodden, 2020. "Cutthroat Capitalism versus Cuddly Socialism: Are Americans More Meritocratic and Efficiency-Seeking than Scandinavians?," Journal of Political Economy, University of Chicago Press, vol. 128(5), pages 1753-1788.
    9. Stefano Dellavigna & John A. List & Ulrike Malmendier & Gautam Rao, 2017. "Voting to Tell Others," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 84(1), pages 143-181.
    10. Binmore, K & Shaked, A & Sutton, J, 1985. "Testing Noncooperative Bargaining Theory: A Preliminary Study," American Economic Review, American Economic Association, vol. 75(5), pages 1178-1180, December.
    11. Kahneman, Daniel & Knetsch, Jack L & Thaler, Richard H, 1986. "Fairness and the Assumptions of Economics," The Journal of Business, University of Chicago Press, vol. 59(4), pages 285-300, October.
    12. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    13. Eran Shmaya & Leeat Yariv, 2016. "Experiments on Decisions under Uncertainty: A Theoretical Framework," American Economic Review, American Economic Association, vol. 106(7), pages 1775-1801, July.
    14. Aviv Nevo & Adam M. Rosen, 2012. "Identification With Imperfect Instruments," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 659-671, August.
    15. John A. List, 2006. "The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions," Journal of Political Economy, University of Chicago Press, vol. 114(1), pages 1-37, February.
    16. Guido W. Imbens & Charles F. Manski, 2004. "Confidence Intervals for Partially Identified Parameters," Econometrica, Econometric Society, vol. 72(6), pages 1845-1857, November.
    17. Gerber, Alan S. & Green, Donald P. & Larimer, Christopher W., 2008. "Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment," American Political Science Review, Cambridge University Press, vol. 102(1), pages 33-48, February.
    18. James Andreoni & Charles Sprenger, 2012. "Estimating Time Preferences from Convex Budgets," American Economic Review, American Economic Association, vol. 102(7), pages 3333-3356, December.
    19. John A. List & Robert P. Berrens & Alok K. Bohara & Joe Kerkvliet, 2004. "Examining the Role of Social Isolation on Stated Preferences," American Economic Review, American Economic Association, vol. 94(3), pages 741-752, June.
    20. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    21. Hoffman, Elizabeth & McCabe, Kevin A & Smith, Vernon L, 1996. "On Expectations and the Monetary Stakes in Ultimatum Games," International Journal of Game Theory, Springer;Game Theory Society, vol. 25(3), pages 289-301.
    22. Nicholas Bardsley, 2008. "Dictator game giving: altruism or artefact?," Experimental Economics, Springer;Economic Science Association, vol. 11(2), pages 122-133, June.
    23. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    24. Thaler, Richard H, 1988. "The Ultimatum Game," Journal of Economic Perspectives, American Economic Association, vol. 2(4), pages 195-206, Fall.
    25. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    26. repec:feb:framed:0087 is not listed on IDEAS
    27. Guth, Werner & Schmittberger, Rolf & Schwarze, Bernd, 1982. "An experimental analysis of ultimatum bargaining," Journal of Economic Behavior & Organization, Elsevier, vol. 3(4), pages 367-388, December.
    28. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    29. Rachel Croson & Uri Gneezy, 2009. "Gender Differences in Preferences," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 448-474, June.
    30. Berg Joyce & Dickhaut John & McCabe Kevin, 1995. "Trust, Reciprocity, and Social History," Games and Economic Behavior, Elsevier, vol. 10(1), pages 122-142, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Emin Karagözoğlu & Ümit Barış Urhan, 2017. "The Effect of Stake Size in Experimental Bargaining and Distribution Games: A Survey," Group Decision and Negotiation, Springer, vol. 26(2), pages 285-325, March.
    3. Weimann Joachim, 2015. "Die Rolle von Verhaltensökonomik und experimenteller Forschung in Wirtschaftswissenschaft und Politikberatung," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 16(3), pages 231-252, October.
    4. Bharat Chandar & Uri Gneezy & John A. List & Ian Muir, 2019. "The Drivers of Social Preferences: Evidence from a Nationwide Tipping Field Experiment," NBER Working Papers 26380, National Bureau of Economic Research, Inc.
    5. Carlos Alós-Ferrer & Jaume García-Segarra & Alexander Ritschel, 2018. "The Big Robber Game," ECON - Working Papers 291, Department of Economics - University of Zurich.
    6. Aycinena, Diego & Bogliacino, Francesco & Kimbrough, Erik O., 2024. "Measuring norms: Assessing the threat of social desirability bias to the Bicchieri and Xiao elicitation method," Journal of Economic Behavior & Organization, Elsevier, vol. 222(C), pages 225-239.
    7. Larney, Andrea & Rotella, Amanda & Barclay, Pat, 2019. "Stake size effects in ultimatum game and dictator game offers: A meta-analysis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 151(C), pages 61-72.
    8. Brit Grosskopf & Graeme Pearce, 2020. "Do You Mind Me Paying Less? Measuring Other-Regarding Preferences in the Market for Taxis," Management Science, INFORMS, vol. 66(11), pages 5059-5074, November.
    9. Walkowitz, Gari, 2019. "On the Validity of Probabilistic (and Cost-Saving) Incentives in Dictator Games: A Systematic Test," MPRA Paper 91541, University Library of Munich, Germany.
    10. Rami Zwick & Xiao-Ping Chen, 1999. "What Price Fairness? A Bargaining Study," Management Science, INFORMS, vol. 45(6), pages 804-823, June.
    11. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    12. Thomas Buser, 2010. "Handedness predicts Social Preferences: Evidence connecting the Lab to the Field," Tinbergen Institute Discussion Papers 10-119/3, Tinbergen Institute.
    13. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    14. Walkowitz, Gari, 2021. "Dictator game variants with probabilistic (and cost-saving) payoffs: A systematic test," Journal of Economic Psychology, Elsevier, vol. 85(C).
    15. Burnham, Terence C., 2013. "Toward a neo-Darwinian synthesis of neoclassical and behavioral economics," Journal of Economic Behavior & Organization, Elsevier, vol. 90(S), pages 113-127.
    16. Pin, Paolo & Rotesi, Tiziano, 2023. "App-based experiments," Journal of Economic Psychology, Elsevier, vol. 99(C).
    17. Walkowitz, Gari, 2017. "On the Validity of Cost-Saving Methods in Dictator-Game Experiments: A Systematic Test," MPRA Paper 83309, University Library of Munich, Germany.
    18. Fredrik Carlsson & Haoran He & Peter Martinsson, 2013. "Easy come, easy go," Experimental Economics, Springer;Economic Science Association, vol. 16(2), pages 190-207, June.
    19. Mehmood, Sultan & Naseer, Shaheen & Chen, Daniel L., 2024. "Altruism in governance: Insights from randomized training for Pakistan's junior ministers," Journal of Development Economics, Elsevier, vol. 170(C).
    20. Murnighan, J. Keith & Wang, Long, 2016. "The social world as an experimental game," Organizational Behavior and Human Decision Processes, Elsevier, vol. 136(C), pages 80-94.

    More about this item

    Keywords

    experimenter demand; beliefs; bounding;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_6516. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.