IDEAS home Printed from https://ideas.repec.org/a/aea/aecrev/v108y2018i11p3266-3302.html
   My bibliography  Save this article

Measuring and Bounding Experimenter Demand

Author

Listed:
  • Jonathan de Quidt
  • Johannes Haushofer
  • Christopher Roth

Abstract

We propose a technique for assessing robustness to demand effects of findings from experiments and surveys. The core idea is that by deliberately inducing demand in a structured way we can bound its influence. We present a model in which participants respond to their beliefs about the researcher's objectives. Bounds are obtained by manipulating those beliefs with "demand treatments." We apply the method to 11 classic tasks, and estimate bounds averaging 0.13 standard deviations, suggesting that typical demand effects are probably modest. We also show how to compute demand-robust treatment effects and how to structurally estimate the model.

Suggested Citation

  • Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2018. "Measuring and Bounding Experimenter Demand," American Economic Review, American Economic Association, vol. 108(11), pages 3266-3302, November.
    • Handle: RePEc:aea:aecrev:v:108:y:2018:i:11:p:3266-3302
    Note: DOI: 10.1257/aer.20171330
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/doi/10.1257/aer.20171330
    Download Restriction: no
    File URL: https://www.aeaweb.org/articles/attachments?retrieve=UcqoUDxaB4iMQIrGoMSU_VdMcKuG3Wko
    Download Restriction: no
    File URL: https://www.aeaweb.org/articles/attachments?retrieve=zHeZK0cQO5i4Mfk-5j3ynET3rfSjwCMU
    Download Restriction: no
    File URL: https://www.aeaweb.org/articles/attachments?retrieve=9_DoZMSSjmAXhFBu6ohVK9gEJFOGGm0_
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Barmettler, Franziska & Fehr, Ernst & Zehnder, Christian, 2012. "Big experimenter is watching you! Anonymity and prosocial behavior in the laboratory," Games and Economic Behavior, Elsevier, vol. 75(1), pages 17-34.
    2. Aurelio Bruzzo, 2018. "Recenti iniziative europee ed italiane per la valorizzazione del patrimonio culturale," Working Papers 2018127, University of Ferrara, Department of Economics.
    3. Stefano DellaVigna & Devin Pope, 2018. "What Motivates Effort? Evidence and Expert Forecasts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(2), pages 1029-1069.
    4. Stefano DellaVigna & John A. List & Ulrike Malmendier, 2012. "Testing for Altruism and Social Pressure in Charitable Giving," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(1), pages 1-56.
    5. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 631-645.
    6. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 151-184, February.
    7. Urs Fischbacher & Franziska Föllmi-Heusi, 2013. "Lies In Disguise—An Experimental Study On Cheating," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 525-547, June.
    8. Ingvild Almås & Alexander W. Cappelen & Bertil Tungodden, 2020. "Cutthroat Capitalism versus Cuddly Socialism: Are Americans More Meritocratic and Efficiency-Seeking than Scandinavians?," Journal of Political Economy, University of Chicago Press, vol. 128(5), pages 1753-1788.
    9. Stefano Dellavigna & John A. List & Ulrike Malmendier & Gautam Rao, 2017. "Voting to Tell Others," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 84(1), pages 143-181.
    10. Binmore, K & Shaked, A & Sutton, J, 1985. "Testing Noncooperative Bargaining Theory: A Preliminary Study," American Economic Review, American Economic Association, vol. 75(5), pages 1178-1180, December.
    11. Kahneman, Daniel & Knetsch, Jack L & Thaler, Richard H, 1986. "Fairness and the Assumptions of Economics," The Journal of Business, University of Chicago Press, vol. 59(4), pages 285-300, October.
    12. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    13. Eran Shmaya & Leeat Yariv, 2016. "Experiments on Decisions under Uncertainty: A Theoretical Framework," American Economic Review, American Economic Association, vol. 106(7), pages 1775-1801, July.
    14. Aviv Nevo & Adam M. Rosen, 2012. "Identification With Imperfect Instruments," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 659-671, August.
    15. John A. List, 2006. "The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions," Journal of Political Economy, University of Chicago Press, vol. 114(1), pages 1-37, February.
    16. Guido W. Imbens & Charles F. Manski, 2004. "Confidence Intervals for Partially Identified Parameters," Econometrica, Econometric Society, vol. 72(6), pages 1845-1857, November.
    17. Gerber, Alan S. & Green, Donald P. & Larimer, Christopher W., 2008. "Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment," American Political Science Review, Cambridge University Press, vol. 102(1), pages 33-48, February.
    18. James Andreoni & Charles Sprenger, 2012. "Estimating Time Preferences from Convex Budgets," American Economic Review, American Economic Association, vol. 102(7), pages 3333-3356, December.
    19. John A. List & Robert P. Berrens & Alok K. Bohara & Joe Kerkvliet, 2004. "Examining the Role of Social Isolation on Stated Preferences," American Economic Review, American Economic Association, vol. 94(3), pages 741-752, June.
    20. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    21. Hoffman, Elizabeth & McCabe, Kevin A & Smith, Vernon L, 1996. "On Expectations and the Monetary Stakes in Ultimatum Games," International Journal of Game Theory, Springer;Game Theory Society, vol. 25(3), pages 289-301.
    22. Nicholas Bardsley, 2008. "Dictator game giving: altruism or artefact?," Experimental Economics, Springer;Economic Science Association, vol. 11(2), pages 122-133, June.
    23. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    24. Thaler, Richard H, 1988. "The Ultimatum Game," Journal of Economic Perspectives, American Economic Association, vol. 2(4), pages 195-206, Fall.
    25. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    26. repec:feb:framed:0087 is not listed on IDEAS
    27. Guth, Werner & Schmittberger, Rolf & Schwarze, Bernd, 1982. "An experimental analysis of ultimatum bargaining," Journal of Economic Behavior & Organization, Elsevier, vol. 3(4), pages 367-388, December.
    28. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    29. Rachel Croson & Uri Gneezy, 2009. "Gender Differences in Preferences," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 448-474, June.
    30. Berg Joyce & Dickhaut John & McCabe Kevin, 1995. "Trust, Reciprocity, and Social History," Games and Economic Behavior, Elsevier, vol. 10(1), pages 122-142, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Emin Karagözoğlu & Ümit Barış Urhan, 2017. "The Effect of Stake Size in Experimental Bargaining and Distribution Games: A Survey," Group Decision and Negotiation, Springer, vol. 26(2), pages 285-325, March.
    3. Weimann Joachim, 2015. "Die Rolle von Verhaltensökonomik und experimenteller Forschung in Wirtschaftswissenschaft und Politikberatung," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 16(3), pages 231-252, October.
    4. Bharat Chandar & Uri Gneezy & John A. List & Ian Muir, 2019. "The Drivers of Social Preferences: Evidence from a Nationwide Tipping Field Experiment," NBER Working Papers 26380, National Bureau of Economic Research, Inc.
    5. Carlos Alós-Ferrer & Jaume García-Segarra & Alexander Ritschel, 2018. "The Big Robber Game," ECON - Working Papers 291, Department of Economics - University of Zurich.
    6. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    7. Larney, Andrea & Rotella, Amanda & Barclay, Pat, 2019. "Stake size effects in ultimatum game and dictator game offers: A meta-analysis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 151(C), pages 61-72.
    8. Brit Grosskopf & Graeme Pearce, 2020. "Do You Mind Me Paying Less? Measuring Other-Regarding Preferences in the Market for Taxis," Management Science, INFORMS, vol. 66(11), pages 5059-5074, November.
    9. Rami Zwick & Xiao-Ping Chen, 1999. "What Price Fairness? A Bargaining Study," Management Science, INFORMS, vol. 45(6), pages 804-823, June.
    10. Walkowitz, Gari, 2017. "On the Validity of Cost-Saving Methods in Dictator-Game Experiments: A Systematic Test," MPRA Paper 83309, University Library of Munich, Germany.
    11. Walkowitz, Gari, 2019. "On the Validity of Probabilistic (and Cost-Saving) Incentives in Dictator Games: A Systematic Test," MPRA Paper 91541, University Library of Munich, Germany.
    12. Fredrik Carlsson & Haoran He & Peter Martinsson, 2013. "Easy come, easy go," Experimental Economics, Springer;Economic Science Association, vol. 16(2), pages 190-207, June.
    13. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    14. Thomas Buser, 2010. "Handedness predicts Social Preferences: Evidence connecting the Lab to the Field," Tinbergen Institute Discussion Papers 10-119/3, Tinbergen Institute.
    15. Murnighan, J. Keith & Wang, Long, 2016. "The social world as an experimental game," Organizational Behavior and Human Decision Processes, Elsevier, vol. 136(C), pages 80-94.
    16. Walkowitz, Gari, 2021. "Dictator game variants with probabilistic (and cost-saving) payoffs: A systematic test," Journal of Economic Psychology, Elsevier, vol. 85(C).
    17. Ananish Chaudhuri & Vegard Iversen & Francesca R. Jensenius & Pushkar Maitra, 2020. "Time in Office and the Changing Gender Gap in Dishonesty: Evidence from Local Politics in India," CESifo Working Paper Series 8217, CESifo.
    18. Burnham, Terence C., 2013. "Toward a neo-Darwinian synthesis of neoclassical and behavioral economics," Journal of Economic Behavior & Organization, Elsevier, vol. 90(S), pages 113-127.
    19. Burks, Stephen V. & Carpenter, Jeffrey P. & Verhoogen, Eric, 2003. "Playing both roles in the trust game," Journal of Economic Behavior & Organization, Elsevier, vol. 51(2), pages 195-216, June.
    20. Stefano DellaVigna, 2009. "Psychology and Economics: Evidence from the Field," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 315-372, June.

    More about this item

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • D91 - Microeconomics - - Micro-Based Behavioral Economics - - - Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:aecrev:v:108:y:2018:i:11:p:3266-3302. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.