IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/17957.html
   My bibliography  Save this paper

On the Generalizability of Experimental Results in Economics

Author

Listed:
  • Omar Al-Ubaydli
  • John A. List

Abstract

Economists are increasingly turning to the experimental method as a means to estimate causal effects. By using randomization to identify key treatment effects, theories previously viewed as untestable are now scrutinized, efficacy of public policies are now more easily verified, and stakeholders can swiftly add empirical evidence to aid their decision-making. This study provides an overview of experimental methods in economics, with a special focus on developing an economic theory of generalizability. Given that field experiments are in their infancy, our secondary focus pertains to a discussion of the various parameters that they identify, and how they add to scientific knowledge. We conclude that until we conduct more field experiments that build a bridge between the lab and the naturally-occurring settings of interest we cannot begin to make strong conclusions empirically on the crucial question of generalizability from the lab to the field.

Suggested Citation

  • Omar Al-Ubaydli & John A. List, 2012. "On the Generalizability of Experimental Results in Economics," NBER Working Papers 17957, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:17957
    Note: PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w17957.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bohm, Peter, 1972. "Estimating demand for public goods: An experiment," European Economic Review, Elsevier, vol. 3(2), pages 111-130.
    2. Ernst Fehr & Klaus M. Schmidt, 1999. "A Theory of Fairness, Competition, and Cooperation," The Quarterly Journal of Economics, Oxford University Press, vol. 114(3), pages 817-868.
    3. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-793, September.
    4. Daniel Rondeau & John List, 2008. "Matching and challenge gifts to charity: evidence from laboratory and natural field experiments," Experimental Economics, Springer;Economic Science Association, vol. 11(3), pages 253-267, September.
    5. Kenneth I. Wolpin & Mark R. Rosenzweig, 2000. "Natural "Natural Experiments" in Economics," Journal of Economic Literature, American Economic Association, vol. 38(4), pages 827-874, December.
    6. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    7. repec:spr:portec:v:1:y:2002:i:2:d:10.1007_s10258-002-0010-3 is not listed on IDEAS
    8. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    9. John A. List, 2006. "The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions," Journal of Political Economy, University of Chicago Press, vol. 114(1), pages 1-37, February.
    10. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    11. Glenn W. Harrison & Morten I. Lau & E. Elisabet Rutström, 2007. "Estimating Risk Attitudes in Denmark: A Field Experiment," Scandinavian Journal of Economics, Wiley Blackwell, vol. 109(2), pages 341-368, June.
    12. Vernon L. Smith, 1962. "An Experimental Study of Competitive Market Behavior," Journal of Political Economy, University of Chicago Press, vol. 70, pages 322-322.
    13. John A. List, 2011. "The Market for Charitable Giving," Journal of Economic Perspectives, American Economic Association, vol. 25(2), pages 157-180, Spring.
    14. John A. List, 2003. "Does Market Experience Eliminate Market Anomalies?," The Quarterly Journal of Economics, Oxford University Press, vol. 118(1), pages 41-71.
    15. repec:feb:natura:0053 is not listed on IDEAS
    16. John A. List, 2004. "Neoclassical Theory Versus Prospect Theory: Evidence from the Marketplace," Econometrica, Econometric Society, vol. 72(2), pages 615-625, March.
    17. Kahneman, Daniel & Knetsch, Jack L & Thaler, Richard, 1986. "Fairness as a Constraint on Profit Seeking: Entitlements in the Market," American Economic Review, American Economic Association, vol. 76(4), pages 728-741, September.
    18. Muriel Niederle & Carmit Segal & Lise Vesterlund, 2013. "How Costly Is Diversity? Affirmative Action in Light of Gender Differences in Competitiveness," Management Science, INFORMS, vol. 59(1), pages 1-16, May.
    19. repec:feb:artefa:0087 is not listed on IDEAS
    20. John A. List, 2011. "Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 3-16, Summer.
    21. Henrich, Joseph & Boyd, Robert & Bowles, Samuel & Camerer, Colin & Fehr, Ernst & Gintis, Herbert (ed.), 2004. "Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-Scale Societies," OUP Catalogue, Oxford University Press, number 9780199262052.
    22. Peter Bohm, 1972. "Estimating the demand for public goods: An experiment," Framed Field Experiments 00126, The Field Experiments Website.
    23. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    24. Phelps, Edmund S, 1972. "The Statistical Theory of Racism and Sexism," American Economic Review, American Economic Association, vol. 62(4), pages 659-661, September.
    25. repec:feb:artefa:0110 is not listed on IDEAS
    26. James J. Heckman, 2000. "Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, Oxford University Press, vol. 115(1), pages 45-97.
    27. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Francis Larson & John List & Robert Metcalfe, 2016. "Can Myopic Loss Aversion Explain the Equity Premium Puzzle? Evidence from a Natural Field Experiment with Professional Traders," Natural Field Experiments 00534, The Field Experiments Website.
    2. Egebark, Johan & Ekström, Mathias, 2016. "Can indifference make the world greener?," Journal of Environmental Economics and Management, Elsevier, vol. 76(C), pages 1-13.
    3. Omar Al-Ubaydli & John List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," Artefactual Field Experiments 00458, The Field Experiments Website.
    4. Decio Coviello & Andrea Guglielmo & Giancarlo Spagnolo, 2015. "The Effect of Discretion on Procurement Performance," CEIS Research Paper 361, Tor Vergata University, CEIS, revised 17 Nov 2015.
    5. Beck, Adrian & Kerschbamer, Rudolf & Qiu, Jianying & Sutter, Matthias, 2014. "Car mechanics in the lab––Investigating the behavior of real experts on experimental markets for credence goods," Journal of Economic Behavior & Organization, Elsevier, vol. 108(C), pages 166-173.
    6. John A. List, 2014. "Using Field Experiments to Change the Template of How We Teach Economics," The Journal of Economic Education, Taylor & Francis Journals, vol. 45(2), pages 81-89, June.
    7. Holmen, Martin & Kirchler, Michael & Kleinlercher, Daniel, 2014. "Do option-like incentives induce overvaluation? Evidence from experimental asset markets," Journal of Economic Dynamics and Control, Elsevier, vol. 40(C), pages 179-194.
    8. Goeschl, Timo & Kettner, Sara Elisa & Lohse, Johannes & Schwieren, Christiane, 2015. "What do we learn from public good games about voluntary climate action? Evidence from an artefactual field experiment," Working Papers 0595, University of Heidelberg, Department of Economics.
    9. Omar Al-Ubaydli & John A. List, 2013. "On the Generalizability of Experimental Results in Economics: With a Response to Commentors," CESifo Working Paper Series 4543, CESifo Group Munich.
    10. Galizzi, Matteo M. & Navarro-Martínez, Daniel, 2018. "On the external validity of social preference games: a systematic lab-field study," LSE Research Online Documents on Economics 84088, London School of Economics and Political Science, LSE Library.
    11. Judd Kessler, 2013. "When will there be Gift Exchange? Addressing the Lab-Field Debate with Laboratory Gift Exchange Experiments," CESifo Working Paper Series 4161, CESifo Group Munich.
    12. repec:aea:jecper:v:31:y:2017:i:4:p:125-44 is not listed on IDEAS
    13. Omar Al-Ubaydli & John A. List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," NBER Working Papers 19666, National Bureau of Economic Research, Inc.
    14. Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2013. "Opting-in: Participation bias in economic experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 43-70.
    15. Adnane Kendel & Nathalie Lazaric & Kevin Maréchal, 2017. "What Do People 'Learn By Looking' at Direct Feedback on their Energy Consumption? Results of a Field Study in Southern France," GREDEG Working Papers 2017-19, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), University of Nice Sophia Antipolis.
    16. repec:eee:enepol:v:108:y:2017:i:c:p:593-605 is not listed on IDEAS
    17. Omar Al-Ubaydli & John A. List & Danielle LoRe & Dana Suskind, 2017. "Scaling for Economists: Lessons from the Non-Adherence Problem in the Medical Literature," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 125-144, Fall.
    18. Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2012. "Opting-In: Participation Biases in the Lab," IZA Discussion Papers 6865, Institute for the Study of Labor (IZA).
    19. Rodney J. Andrews & Trevon D. Logan & Michael J. Sinkey, 2012. "Identifying Confirmatory Bias in the Field: Evidence from a Poll of Experts," NBER Working Papers 18064, National Bureau of Economic Research, Inc.

    More about this item

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D03 - Microeconomics - - General - - - Behavioral Microeconomics: Underlying Principles

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:17957. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: () or (Joanne Lustig). General contact details of provider: http://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.