IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/75461.html
   My bibliography  Save this paper

Evaluating replicability of laboratory experiments in Economics

Author

Listed:
  • Camerer, Colin
  • Dreber, Anna
  • Forsell, Eskil
  • Ho, Teck-Hua
  • Huber, Jurgen
  • Johannesson, Magnus
  • Kirchler, Michael
  • Almenberg, Johan
  • Altmejd, Adam
  • Chan, Taizan
  • Heikensten, Emma
  • Holzmeister, Felix
  • Imai, Taisuke
  • Isaksson, Siri
  • Nave, Gideon
  • Pfeiffer, Thomas
  • Razen, Michael
  • Wu, Hang

Abstract

The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.

Suggested Citation

  • Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:75461
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/75461/1/MPRA_paper_75461.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Manski, Charles F., 2006. "Interpreting the predictions of prediction markets," Economics Letters, Elsevier, vol. 91(3), pages 425-429, June.
    2. Berg, Joyce & Forsythe, Robert & Nelson, Forrest & Rietz, Thomas, 2008. "Results from a Dozen Years of Election Futures Markets Research," Handbook of Experimental Economics Results, in: Charles R. Plott & Vernon L. Smith (ed.), Handbook of Experimental Economics Results, edition 1, volume 1, chapter 80, pages 742-751, Elsevier.
    3. Johannes Abeler & Armin Falk & Lorenz Goette & David Huffman, 2011. "Reference Points and Effort Provision," American Economic Review, American Economic Association, vol. 101(2), pages 470-492, April.
    4. Bjorn Bartling & Ernst Fehr & Klaus M. Schmidt, 2012. "Screening, Competition, and Job Design: Economic Origins of Good Jobs," American Economic Review, American Economic Association, vol. 102(2), pages 834-864, April.
    5. Christian Franz Horn & Bjoern Sven Ivens & Michael Ohneberg & Alexander Brem, 2014. "Ideas Markets: Prediction Markets – A literature review 2014," Journal of Prediction Markets, University of Buckingham Press, vol. 8(2), pages 89-126.
    6. Shimon Kogan & Anthony M. Kwasnica & Roberto A. Weber, 2011. "Coordination in the Presence of Asset Markets," American Economic Review, American Economic Association, vol. 101(2), pages 927-947, April.
    7. Ernst Fehr & Holger Herz & Tom Wilkening, 2013. "The Lure of Authority: Motivation and Incentive Effects of Power," American Economic Review, American Economic Association, vol. 103(4), pages 1325-1359, June.
    8. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    9. David Gill & Victoria Prowse, 2012. "A Structural Analysis of Disappointment Aversion in a Real Effort Competition," American Economic Review, American Economic Association, vol. 102(1), pages 469-503, February.
    10. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    11. Catherine C. Eckel & Ragan Petrie, 2011. "Face Value," American Economic Review, American Economic Association, vol. 101(4), pages 1497-1513, June.
    12. John Ifcher & Homa Zarghamee, 2011. "Happiness and Time Preference: The Effect of Positive Affect in a Random-Assignment Experiment," American Economic Review, American Economic Association, vol. 101(7), pages 3109-3129, December.
    13. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    14. Geoffroy de Clippel & Kfir Eliaz & Brian Knight, 2014. "On the Selection of Arbitrators," American Economic Review, American Economic Association, vol. 104(11), pages 3434-3458, November.
    15. Attila Ambrus & Ben Greiner, 2012. "Imperfect Public Monitoring with Costly Punishment: An Experimental Study," American Economic Review, American Economic Association, vol. 102(7), pages 3317-3332, December.
    16. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    17. Drew Fudenberg & David G. Rand & Anna Dreber, 2012. "Slow to Anger and Fast to Forgive: Cooperation in an Uncertain World," American Economic Review, American Economic Association, vol. 102(2), pages 720-749, April.
    18. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    19. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    20. Robin Hanson, 2007. "Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 3-15, February.
    21. Ryan W. Buell & Taly Reich & Michael I. Norton, 2014. ""Last-Place Aversion": Evidence and Redistributive Implications," The Quarterly Journal of Economics, Oxford University Press, vol. 129(1), pages 105-149.
    22. Judd B. Kessler & Alvin E. Roth, 2012. "Organ Allocation Policy and the Decision to Donate," American Economic Review, American Economic Association, vol. 102(5), pages 2018-2047, August.
    23. Steffen Huck & Andrew J. Seltzer & Brian Wallace, 2011. "Deferred Compensation in Multiperiod Labor Contracts: An Experimental Test of Lazear's Model," American Economic Review, American Economic Association, vol. 101(2), pages 819-843, April.
    24. Daniel Friedman & Ryan Oprea, 2012. "A Continuous Dilemma," American Economic Review, American Economic Association, vol. 102(1), pages 337-363, February.
    25. Nisvan Erkal & Lata Gangadharan & Nikos Nikiforakis, 2011. "Relative Earnings and Giving in a Real-Effort Experiment," American Economic Review, American Economic Association, vol. 101(7), pages 3330-3348, December.
    26. Uwe Dulleck & Rudolf Kerschbamer & Matthias Sutter, 2011. "The Economics of Credence Goods: An Experiment on the Role of Liability, Verifiability, Reputation, and Competition," American Economic Review, American Economic Association, vol. 101(2), pages 526-555, April.
    27. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    28. Keith M. Marzilli Ericson & Andreas Fuster, 2011. "Expectations as Endowments: Evidence on Reference-Dependent Preferences from Exchange and Valuation Experiments," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1879-1907.
    29. Georgios Tziralis & Ilias Tatsiopoulos, 2007. "Prediction Markets: An Extended Literature Review," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 75-91, February.
    30. Roy Chen & Yan Chen, 2011. "The Potential of Social Identity for Equilibrium Selection," American Economic Review, American Economic Association, vol. 101(6), pages 2562-2589, October.
    31. Roth, Alvin E, 1994. "Lets Keep the Con out of Experimental Econ.: A Methodological Note," Empirical Economics, Springer, vol. 19(2), pages 279-289.
    32. Michael Kirchler & Jurgen Huber & Thomas Stockl, 2012. "Thar She Bursts: Reducing Confusion Reduces Bubbles," American Economic Review, American Economic Association, vol. 102(2), pages 865-883, April.
    33. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    34. Brian Merlob & Charles R. Plott & Yuanjun Zhang, 2012. "The CMS Auction: Experimental Studies of a Median-Bid Procurement Auction with Nonbinding Bids," The Quarterly Journal of Economics, Oxford University Press, vol. 127(2), pages 793-827.
    35. John Duffy & Daniela Puzzello, 2014. "Gift Exchange versus Monetary Exchange: Theory and Evidence," American Economic Review, American Economic Association, vol. 104(6), pages 1735-1776, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    2. Snowberg, Erik & Wolfers, Justin & Zitzewitz, Eric, 2013. "Prediction Markets for Economic Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 657-687, Elsevier.
    3. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    4. Mikuláš Gangur & Miroslav Plevný, 2014. "Tools for Consumer Rights Protection in the Prediction of Electronic Virtual Market and Technological Changes," The AMFITEATRU ECONOMIC journal, Academy of Economic Studies - Bucharest, Romania, vol. 16(36), pages 578-578, May.
    5. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    6. Gill, David & Prowse, Victoria, 2019. "Measuring costly effort using the slider task," Journal of Behavioral and Experimental Finance, Elsevier, vol. 21(C), pages 1-9.
    7. Khan, Urmee & Lieli, Robert P., 2018. "Information flow between prediction markets, polls and media: Evidence from the 2008 presidential primaries," International Journal of Forecasting, Elsevier, vol. 34(4), pages 696-710.
    8. Stefan Palan & Jürgen Huber & Larissa Senninger, 2020. "Aggregation mechanisms for crowd predictions," Experimental Economics, Springer;Economic Science Association, vol. 23(3), pages 788-814, September.
    9. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    10. Dieckmann, Anja & Grimm, Veronika & Unfried, Matthias & Utikal, Verena & Valmasoni, Lorenzo, 2016. "On trust in honesty and volunteering among Europeans: Cross-country evidence on perceptions and behavior," European Economic Review, Elsevier, vol. 90(C), pages 225-253.
    11. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    12. Simon Gächter & Lingbo Huang & Martin Sefton, 2018. "Disappointment Aversion And Social Comparisons In A Real‐Effort Competition," Economic Inquiry, Western Economic Association International, vol. 56(3), pages 1512-1525, July.
    13. Dertwinkel-Kalt, Markus & Köster, Mats, 2017. "Salient compromises in the newsvendor game," Journal of Economic Behavior & Organization, Elsevier, vol. 141(C), pages 301-315.
    14. Kewei Hou & Chen Xue & Lu Zhang, 2017. "Replicating Anomalies," NBER Working Papers 23394, National Bureau of Economic Research, Inc.
    15. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    16. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    17. Macera, Rosario, 2018. "Intertemporal incentives under loss aversion," Journal of Economic Theory, Elsevier, vol. 178(C), pages 551-594.
    18. Gall, Thomas & Hu, Xiaocheng & Vlassopoulos, Michael, 2016. "Dynamic Incentive Effects of Team Formation: Experimental Evidence," IZA Discussion Papers 10393, Institute of Labor Economics (IZA).
    19. Rosato, Antonio & Tymula, Agnieszka A., 2019. "Loss aversion and competition in Vickrey auctions: Money ain't no good," Games and Economic Behavior, Elsevier, vol. 115(C), pages 188-208.
    20. Brocas, Isabelle & Carrillo, Juan D. & Giga, Aleksandar & Zapatero, Fernando, 2019. "Risk Aversion in a Dynamic Asset Allocation Experiment," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 54(5), pages 2209-2232, October.

    More about this item

    Keywords

    replication; economics; experiment;
    All these keywords.

    JEL classification:

    • A12 - General Economics and Teaching - - General Economics - - - Relation of Economics to Other Disciplines
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:75461. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.