IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/75461.html
   My bibliography  Save this paper

Evaluating replicability of laboratory experiments in Economics

Author

Listed:
  • Camerer, Colin
  • Dreber, Anna
  • Forsell, Eskil
  • Ho, Teck-Hua
  • Huber, Jurgen
  • Johannesson, Magnus
  • Kirchler, Michael
  • Almenberg, Johan
  • Altmejd, Adam
  • Chan, Taizan
  • Heikensten, Emma
  • Holzmeister, Felix
  • Imai, Taisuke
  • Isaksson, Siri
  • Nave, Gideon
  • Pfeiffer, Thomas
  • Razen, Michael
  • Wu, Hang

Abstract

The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.

Suggested Citation

  • Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:75461
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/75461/1/MPRA_paper_75461.pdf
    File Function: original version
    Download Restriction: no

    References listed on IDEAS

    as
    1. Manski, Charles F., 2006. "Interpreting the predictions of prediction markets," Economics Letters, Elsevier, vol. 91(3), pages 425-429, June.
    2. Berg, Joyce & Forsythe, Robert & Nelson, Forrest & Rietz, Thomas, 2008. "Results from a Dozen Years of Election Futures Markets Research," Handbook of Experimental Economics Results, Elsevier.
    3. Johannes Abeler & Armin Falk & Lorenz Goette & David Huffman, 2011. "Reference Points and Effort Provision," American Economic Review, American Economic Association, vol. 101(2), pages 470-492, April.
    4. Bjorn Bartling & Ernst Fehr & Klaus M. Schmidt, 2012. "Screening, Competition, and Job Design: Economic Origins of Good Jobs," American Economic Review, American Economic Association, vol. 102(2), pages 834-864, April.
    5. Christian Franz Horn & Bjoern Sven Ivens & Michael Ohneberg & Alexander Brem, 2014. "Ideas Markets: Prediction Markets – A literature review 2014," Journal of Prediction Markets, University of Buckingham Press, vol. 8(2), pages 89-126.
    6. Shimon Kogan & Anthony M. Kwasnica & Roberto A. Weber, 2011. "Coordination in the Presence of Asset Markets," American Economic Review, American Economic Association, vol. 101(2), pages 927-947, April.
    7. Ernst Fehr & Holger Herz & Tom Wilkening, 2013. "The Lure of Authority: Motivation and Incentive Effects of Power," American Economic Review, American Economic Association, vol. 103(4), pages 1325-1359, June.
    8. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    9. David Gill & Victoria Prowse, 2012. "A Structural Analysis of Disappointment Aversion in a Real Effort Competition," American Economic Review, American Economic Association, vol. 102(1), pages 469-503, February.
    10. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    11. Catherine C. Eckel & Ragan Petrie, 2011. "Face Value," American Economic Review, American Economic Association, vol. 101(4), pages 1497-1513, June.
    12. John Ifcher & Homa Zarghamee, 2011. "Happiness and Time Preference: The Effect of Positive Affect in a Random-Assignment Experiment," American Economic Review, American Economic Association, vol. 101(7), pages 3109-3129, December.
    13. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    14. Geoffroy de Clippel & Kfir Eliaz & Brian Knight, 2014. "On the Selection of Arbitrators," American Economic Review, American Economic Association, vol. 104(11), pages 3434-3458, November.
    15. Attila Ambrus & Ben Greiner, 2012. "Imperfect Public Monitoring with Costly Punishment: An Experimental Study," American Economic Review, American Economic Association, vol. 102(7), pages 3317-3332, December.
    16. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    17. Drew Fudenberg & David G. Rand & Anna Dreber, 2012. "Slow to Anger and Fast to Forgive: Cooperation in an Uncertain World," American Economic Review, American Economic Association, vol. 102(2), pages 720-749, April.
    18. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    19. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    20. Robin Hanson, 2007. "Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 3-15, February.
    21. Ryan W. Buell & Taly Reich & Michael I. Norton, 2014. ""Last-Place Aversion": Evidence and Redistributive Implications," The Quarterly Journal of Economics, Oxford University Press, vol. 129(1), pages 105-149.
    22. Judd B. Kessler & Alvin E. Roth, 2012. "Organ Allocation Policy and the Decision to Donate," American Economic Review, American Economic Association, vol. 102(5), pages 2018-2047, August.
    23. Steffen Huck & Andrew J. Seltzer & Brian Wallace, 2011. "Deferred Compensation in Multiperiod Labor Contracts: An Experimental Test of Lazear's Model," American Economic Review, American Economic Association, vol. 101(2), pages 819-843, April.
    24. Daniel Friedman & Ryan Oprea, 2012. "A Continuous Dilemma," American Economic Review, American Economic Association, vol. 102(1), pages 337-363, February.
    25. Nisvan Erkal & Lata Gangadharan & Nikos Nikiforakis, 2011. "Relative Earnings and Giving in a Real-Effort Experiment," American Economic Review, American Economic Association, vol. 101(7), pages 3330-3348, December.
    26. Uwe Dulleck & Rudolf Kerschbamer & Matthias Sutter, 2011. "The Economics of Credence Goods: An Experiment on the Role of Liability, Verifiability, Reputation, and Competition," American Economic Review, American Economic Association, vol. 101(2), pages 526-555, April.
    27. Keith M. Marzilli Ericson & Andreas Fuster, 2011. "Expectations as Endowments: Evidence on Reference-Dependent Preferences from Exchange and Valuation Experiments," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1879-1907.
    28. Georgios Tziralis & Ilias Tatsiopoulos, 2007. "Prediction Markets: An Extended Literature Review," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 75-91, February.
    29. Roy Chen & Yan Chen, 2011. "The Potential of Social Identity for Equilibrium Selection," American Economic Review, American Economic Association, vol. 101(6), pages 2562-2589, October.
    30. Roth, Alvin E, 1994. "Lets Keep the Con out of Experimental Econ.: A Methodological Note," Empirical Economics, Springer, vol. 19(2), pages 279-289.
    31. Michael Kirchler & Jurgen Huber & Thomas Stockl, 2012. "Thar She Bursts: Reducing Confusion Reduces Bubbles," American Economic Review, American Economic Association, vol. 102(2), pages 865-883, April.
    32. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    33. Brian Merlob & Charles R. Plott & Yuanjun Zhang, 2012. "The CMS Auction: Experimental Studies of a Median-Bid Procurement Auction with Nonbinding Bids," The Quarterly Journal of Economics, Oxford University Press, vol. 127(2), pages 793-827.
    34. John Duffy & Daniela Puzzello, 2014. "Gift Exchange versus Monetary Exchange: Theory and Evidence," American Economic Review, American Economic Association, vol. 104(6), pages 1735-1776, June.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    replication; economics; experiment;

    JEL classification:

    • A12 - General Economics and Teaching - - General Economics - - - Relation of Economics to Other Disciplines
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:75461. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Joachim Winter). General contact details of provider: http://edirc.repec.org/data/vfmunde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.