IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/25858.html
   My bibliography  Save this paper

Stability of Experimental Results: Forecasts and Evidence

Author

Listed:
  • Stefano DellaVigna
  • Devin Pope

Abstract

How robust are experimental results to changes in design? And can researchers anticipate which changes matter most? We consider a specific context, a real-effort task with multiple behavioral treatments, and examine the stability along six dimensions: (i) pure replication; (ii) demographics; (iii) geography and culture; (iv) the task; (v) the output measure; (vi) the presence of a consent form. We use rank-order correlation across the treatments as measure of stability, and compare the observed correlation to the one under a benchmark of full stability (which allows for noise), and to expert forecasts. The academic experts expect that the pure replication will be close to perfect, that the results will differ sizably across demographic groups (age/gender/education), and that changes to the task and output will make a further impact. We find near perfect replication of the experimental results, and full stability of the results across demographics, significantly higher than the experts expected. The results are quite different across task and output change, mostly because the task change adds noise to the findings. The results are also stable to the lack of consent. Overall, the full stability benchmark is an excellent predictor of the observed stability, while expert forecasts are not that informative. This suggests that researchers' predictions about external validity may not be as informative as they expect. We discuss the implications of both the methods and the results for conceptual replication.

Suggested Citation

  • Stefano DellaVigna & Devin Pope, 2019. "Stability of Experimental Results: Forecasts and Evidence," NBER Working Papers 25858, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:25858
    Note: LS
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w25858.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jordi Brandts & Gary Charness, 2011. "The strategy versus the direct-response method: a first survey of experimental comparisons," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 375-398, September.
    2. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    3. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    4. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    5. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, Oxford University Press, vol. 130(3), pages 1117-1165.
    6. Marianne Bertrand & Dean Karlan & Sendhil Mullainathan & Eldar Shafir & Jonathan Zinman, 2010. "What's Advertising Content Worth? Evidence from a Consumer Credit Marketing Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 125(1), pages 263-306.
    7. Gerber, Alan S. & Green, Donald P., 2000. "The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment," American Political Science Review, Cambridge University Press, vol. 94(3), pages 653-663, September.
    8. Ilyana Kuziemko & Michael I. Norton & Emmanuel Saez & Stefanie Stantcheva, 2015. "How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments," American Economic Review, American Economic Association, vol. 105(4), pages 1478-1508, April.
    9. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    10. Saurabh Bhargava & Dayanand Manoli, 2015. "Psychological Frictions and the Incomplete Take-Up of Social Benefits: Evidence from an IRS Field Experiment," American Economic Review, American Economic Association, vol. 105(11), pages 3489-3529, November.
    11. Felipe A. Araujo & Erin Carbone & Lynn Conell-Price & Marli W. Dunietz & Ania Jaroszewicz & Rachel Landsman & Diego Lamé & Lise Vesterlund & Stephanie W. Wang & Alistair J. Wilson, 2016. "The slider task: an example of restricted inference on incentive effects," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 2(1), pages 1-12, May.
    12. Stefano DellaVigna & Devin Pope, 2018. "What Motivates Effort? Evidence and Expert Forecasts," Review of Economic Studies, Oxford University Press, vol. 85(2), pages 1029-1069.
    13. Erik Snowberg & Leeat Yariv, 2018. "Testing the Waters: Behavior across Participant Pools," NBER Working Papers 24781, National Bureau of Economic Research, Inc.
    14. Uri Gneezy & John A List, 2006. "Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments," Econometrica, Econometric Society, vol. 74(5), pages 1365-1384, September.
    15. Snowberg, Erik & Yariv, Leeat, 2018. "Testing the Waters: Behavior across Participant Pools," CEPR Discussion Papers 13015, C.E.P.R. Discussion Papers.
    16. Erik Snowberg & Leeat Yariv, 2018. "Testing the Waters: Behavior across Participant Pools," CESifo Working Paper Series 7136, CESifo.
    17. Imas, Alex, 2014. "Working for the “warm glow”: On the benefits and limits of prosocial incentives," Journal of Public Economics, Elsevier, vol. 114(C), pages 14-18.
    18. Alan Gerber & Donald Green, 2000. "The effects of canvassing, direct mail, and telephone contact on voter turnout: A field experiment," Natural Field Experiments 00248, The Field Experiments Website.
    19. David Laibson, 1997. "Golden Eggs and Hyperbolic Discounting," The Quarterly Journal of Economics, Oxford University Press, vol. 112(2), pages 443-478.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Gagnon, Nickolas & Bosmans, Kristof & Riedl, Arno, 2020. "The Effect of Unfair Chances and Gender Discrimination on Labor Supply," IZA Discussion Papers 12912, Institute of Labor Economics (IZA).
    3. Stefano DellaVigna & Devin Pope, 2018. "What Motivates Effort? Evidence and Expert Forecasts," Review of Economic Studies, Oxford University Press, vol. 85(2), pages 1029-1069.
    4. Goeschl, Timo & Kettner, Sara Elisa & Lohse, Johannes & Schwieren, Christiane, 2020. "How much can we learn about voluntary climate action from behavior in public goods games?," Ecological Economics, Elsevier, vol. 171(C).
    5. Kaisa Kotakorpi & Satu Metsälampi & Topi Miettinen & Tuomas Nurminen, 2019. "The effect of reporting institutions on tax evasion:Evidence from the lab," Discussion Papers 127, Aboa Centre for Economics.
    6. Gary Bolton & Eugen Dimant & Ulrich Schmidt, 2018. "When a Nudge Backfires. Using Observation with Social and Economic Incentives to Promote Pro-Social Behavior," PPE Working Papers 0017, Philosophy, Politics and Economics, University of Pennsylvania.
    7. S. Dellavigna., 2011. "Psychology and Economics: Evidence from the Field," VOPROSY ECONOMIKI, N.P. Redaktsiya zhurnala "Voprosy Economiki", vol. 4.
    8. Omar Al-Ubaydli & John List, 2016. "Field Experiments in Markets," Artefactual Field Experiments j0002, The Field Experiments Website.
    9. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," NBER Working Papers 25072, National Bureau of Economic Research, Inc.
    10. Cristina Bicchieri & Eugen Dimant & Erte Xiao, 2017. "Deviant or Wrong? The Effects of Norm Information on the Efficacy of Punishment," Discussion Papers 2017-14, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    11. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    12. Dimant, Eugen & van Kleef, Gerben A. & Shalvi, Shaul, 2020. "Requiem for a Nudge: Framing effects in nudging honesty," Journal of Economic Behavior & Organization, Elsevier, vol. 172(C), pages 247-266.
    13. Engelmann, Dirk & Janeba, Eckhard & Mechtenberg, Lydia & Wehrhöfer, Nils, 2019. "Preferences over Taxation of High Income Individuals: Evidence from a Survey Experiment," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203648, Verein für Socialpolitik / German Economic Association.
    14. Hans-Theo Normann & Till Requate & Israel Waichman, 2014. "Do short-term laboratory experiments provide valid descriptions of long-term economic interactions? A study of Cournot markets," Experimental Economics, Springer;Economic Science Association, vol. 17(3), pages 371-390, September.
    15. Andreas C. Drichoutis & Veronika Grimm & Alexandros Karakostas, 2020. "Bribing to Queue-Jump: An experiment on cultural differences in bribing attitudes among Greeks and Germans," Working Papers 2020-2, Agricultural University of Athens, Department Of Agricultural Economics.
    16. Page, Lionel & Sarkar, Dipanwita & Silva-Goncalves, Juliana, 2019. "Long-lasting effects of relative age at school," Journal of Economic Behavior & Organization, Elsevier, vol. 168(C), pages 166-195.
    17. Jonathan Schulz & Uwe Sunde & Petra Thiemann & Christian Thoeni, 2019. "Selection into Experiments: Evidence from a Population of Students," Discussion Papers 2019-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    18. Grant D. Jacobsen, 2019. "How do different sources of policy analysis affect policy preferences? Experimental evidence from the United States," Policy Sciences, Springer;Society of Policy Sciences, vol. 52(3), pages 315-342, September.
    19. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    20. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.

    More about this item

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:25858. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.