IDEAS home Printed from https://ideas.repec.org/p/feb/artefa/00711.html
   My bibliography  Save this paper

Non est Disputandum de Generalizability? A Glimpse into The External Validity Trial

Author

Listed:
  • John List

Abstract

While empirical economics has made important strides over the past half century, there is a recent attack that threatens the foundations of the empirical approach in economics: external validity. Certain dogmatic arguments are not new, yet in some circles the generalizability question is beyond dispute, rendering empirical work as a passive enterprise based on frivolity. Such arguments serve to caution even the staunchest empirical advocates from even starting an empirical inquiry in a novel setting. In its simplest form, questions of external validity revolve around whether the results of the received study can be generalized to different people, situations, stimuli, and time periods. This study clarifies and places the external validity crisis into perspective by taking a unique glimpse into the grandest of trials: The External Validity Trial. A key outcome of the proceedings is an Author Onus Probandi, which outlines four key areas that every study should report to address external validity. Such an evaluative approach properly rewards empirical advances and justly recognizes inherent empirical limitations.

Suggested Citation

  • John List, 2020. "Non est Disputandum de Generalizability? A Glimpse into The External Validity Trial," Artefactual Field Experiments 00711, The Field Experiments Website.
  • Handle: RePEc:feb:artefa:00711
    as

    Download full text from publisher

    File URL: http://s3.amazonaws.com/fieldexperiments-papers2/papers/00711.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, Oxford University Press, vol. 122(3), pages 1235-1264.
    2. Marianne Bertrand & Magne Mogstad & Jack Mountjoy, 2021. "Improving Educational Pathways to Social Mobility: Evidence from Norway’s Reform 94," Journal of Labor Economics, University of Chicago Press, vol. 39(4), pages 965-1010.
    3. List, John A. & Shogren, Jason F., 1998. "Calibration of the difference between actual and hypothetical valuations in a field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 37(2), pages 193-205, October.
    4. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1593-1660.
    5. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    6. Botond Kőszegi & Matthew Rabin, 2006. "A Model of Reference-Dependent Preferences," The Quarterly Journal of Economics, Oxford University Press, vol. 121(4), pages 1133-1165.
    7. Steven D. Levitt & John A. List, 2007. "Viewpoint: On the generalizability of lab behaviour to the field," Canadian Journal of Economics, Canadian Economics Association, vol. 40(2), pages 347-370, May.
    8. List John A., 2007. "Field Experiments: A Bridge between Lab and Naturally Occurring Data," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(2), pages 1-47, April.
    9. Glenn W. Harrison & Morten I. Lau & E. Elisabet Rutström, 2007. "Estimating Risk Attitudes in Denmark: A Field Experiment," Scandinavian Journal of Economics, Wiley Blackwell, vol. 109(2), pages 341-368, June.
    10. Omar Al-Ubaydli & John A. List & Dana Suskind, 2019. "The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments," NBER Working Papers 25848, National Bureau of Economic Research, Inc.
    11. David Lucking-Reiley & John A. List, 2000. "Demand Reduction in Multiunit Auctions: Evidence from a Sportscard Field Experiment," American Economic Review, American Economic Association, vol. 90(4), pages 961-972, September.
    12. Esther Duflo & Emmanuel Saez, 2003. "The Role of Information and Social Interactions in Retirement Plan Decisions: Evidence from a Randomized Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 118(3), pages 815-842.
    13. Claudia Goldin, 2014. "A Grand Gender Convergence: Its Last Chapter," American Economic Review, American Economic Association, vol. 104(4), pages 1091-1119, April.
    14. Roland G. Fryer, Jr. & Steven D. Levitt & John A. List, 2015. "Parental Incentives and Early Childhood Achievement: A Field Experiment in Chicago Heights," NBER Working Papers 21477, National Bureau of Economic Research, Inc.
    15. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: Reply to Kataria," Econ Journal Watch, Econ Journal Watch, vol. 11(1), pages 11-16, January.
    16. Raghabendra Chattopadhyay & Esther Duflo, 2004. "Women as Policy Makers: Evidence from a Randomized Policy Experiment in India," Econometrica, Econometric Society, vol. 72(5), pages 1409-1443, September.
    17. Raj Chetty & Nathaniel Hendren & Patrick Kline & Emmanuel Saez, 2014. "Where is the land of Opportunity? The Geography of Intergenerational Mobility in the United States," The Quarterly Journal of Economics, Oxford University Press, vol. 129(4), pages 1553-1623.
    18. John A. List, 2003. "Does Market Experience Eliminate Market Anomalies?," The Quarterly Journal of Economics, Oxford University Press, vol. 118(1), pages 41-71.
    19. repec:hrv:faseco:30750027 is not listed on IDEAS
    20. John A. List, 2004. "Neoclassical Theory Versus Prospect Theory: Evidence from the Marketplace," Econometrica, Econometric Society, vol. 72(2), pages 615-625, March.
    21. John A. List, 2001. "Do Explicit Warnings Eliminate the Hypothetical Bias in Elicitation Procedures? Evidence from Field Auctions for Sportscards," American Economic Review, American Economic Association, vol. 91(5), pages 1498-1507, December.
    22. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    23. Joseph Henrich & Steve J. Heine & Ara Norenzayan, 2010. "The Weirdest People in the World?," RatSWD Working Papers 139, German Data Forum (RatSWD).
    24. Kahneman, Daniel & Knetsch, Jack L & Thaler, Richard H, 1990. "Experimental Tests of the Endowment Effect and the Coase Theorem," Journal of Political Economy, University of Chicago Press, vol. 98(6), pages 1325-1348, December.
    25. F. H. Knight, 1921. "Cost of Production and Price over Long and Short Periods," Journal of Political Economy, University of Chicago Press, vol. 29, pages 304-304.
    26. Cody Cook & Rebecca Diamond & Jonathan V Hall & John A List & Paul Oyer, 2021. "The Gender Earnings Gap in the Gig Economy: Evidence from over a Million Rideshare Drivers [Measuring the Gig Economy: Current Knowledge and Open Issues]," Review of Economic Studies, Oxford University Press, vol. 88(5), pages 2210-2238.
    27. Omar Al-Ubaydli & John A. List & Dana L. Suskind, 2017. "What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results," American Economic Review, American Economic Association, vol. 107(5), pages 282-286, May.
    28. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    29. Raj Chetty & John N. Friedman & Nathaniel Hendren & Maggie R. Jones & Sonya R. Porter, 2018. "The Opportunity Atlas: Mapping the Childhood Roots of Social Mobility," NBER Working Papers 25147, National Bureau of Economic Research, Inc.
    30. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    31. repec:feb:framed:0052 is not listed on IDEAS
    32. repec:feb:framed:0043 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hallsworth, Michael & List, John A. & Metcalfe, Robert D. & Vlaev, Ivo, 2017. "The behavioralist as tax collector: Using natural field experiments to enhance tax compliance," Journal of Public Economics, Elsevier, vol. 148(C), pages 14-31.
    2. John List, 2021. "2021 Summary Data of Artefactual Field Experiments Published on Fieldexperiments.com," Artefactual Field Experiments 00749, The Field Experiments Website.
    3. John A. List, 2014. "Using Field Experiments to Change the Template of How We Teach Economics," The Journal of Economic Education, Taylor & Francis Journals, vol. 45(2), pages 81-89, June.
    4. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    5. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    6. Jeffrey Flory & Uri Gneezy & Kenneth Leonard & John List, 2012. "Sex, competitiveness, and investment in offspring: On the origin of preferences," Artefactual Field Experiments 00072, The Field Experiments Website.
    7. John List, 2022. "2021 Summary Data of Natural Field Experiments Published on Fieldexperiments.com," Natural Field Experiments 00747, The Field Experiments Website.
    8. List John A., 2007. "Field Experiments: A Bridge between Lab and Naturally Occurring Data," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(2), pages 1-47, April.
    9. Omar Al-Ubaydli & John List, 2016. "Field Experiments in Markets," Artefactual Field Experiments j0002, The Field Experiments Website.
    10. Steven D. Levitt & John A. List, 2007. "Viewpoint: On the generalizability of lab behaviour to the field," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(2), pages 347-370, May.
    11. Omar Al-Ubaydli & John A. List, 2013. "On the Generalizability of Experimental Results in Economics: With a Response to Commentors," CESifo Working Paper Series 4543, CESifo.
    12. Lange, Andreas & Ratan, Anmol, 2010. "Multi-dimensional reference-dependent preferences in sealed-bid auctions - How (most) laboratory experiments differ from the field," Games and Economic Behavior, Elsevier, vol. 68(2), pages 634-645, March.
    13. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    14. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    15. John List & Michael Price, 2013. "Using Field Experiments in Environmental and Resource Economics," Artefactual Field Experiments 00447, The Field Experiments Website.
    16. John List, 2022. "Framed Field Experiments: 2021 Summary on Fieldexperiments.com," Framed Field Experiments 00752, The Field Experiments Website.
    17. S. Dellavigna., 2011. "Psychology and Economics: Evidence from the Field," VOPROSY ECONOMIKI, N.P. Redaktsiya zhurnala "Voprosy Economiki", vol. 5.
    18. Sivan Frenkel & Yuval Heller & Roee Teper, 2018. "The Endowment Effect As Blessing," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 59(3), pages 1159-1186, August.
    19. Johnston, Andrew C., 2021. "Preferences, Selection, and the Structure of Teacher Pay," IZA Discussion Papers 14831, Institute of Labor Economics (IZA).
    20. John A. List & Michael K. Price, 2016. "Editor's Choice The Use of Field Experiments in Environmental and Resource Economics," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 10(2), pages 206-225.

    More about this item

    JEL classification:

    • C01 - Mathematical and Quantitative Methods - - General - - - Econometrics
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D9 - Microeconomics - - Micro-Based Behavioral Economics

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:feb:artefa:00711. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.fieldexperiments.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joe Seidel The email address of this maintainer does not seem to be valid anymore. Please ask Joe Seidel to update the entry or send us the correct address (email available below). General contact details of provider: http://www.fieldexperiments.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.