IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/17957.html
   My bibliography  Save this paper

On the Generalizability of Experimental Results in Economics

Author

Listed:
  • Omar Al-Ubaydli
  • John A. List

Abstract

Economists are increasingly turning to the experimental method as a means to estimate causal effects. By using randomization to identify key treatment effects, theories previously viewed as untestable are now scrutinized, efficacy of public policies are now more easily verified, and stakeholders can swiftly add empirical evidence to aid their decision-making. This study provides an overview of experimental methods in economics, with a special focus on developing an economic theory of generalizability. Given that field experiments are in their infancy, our secondary focus pertains to a discussion of the various parameters that they identify, and how they add to scientific knowledge. We conclude that until we conduct more field experiments that build a bridge between the lab and the naturally-occurring settings of interest we cannot begin to make strong conclusions empirically on the crucial question of generalizability from the lab to the field.

Suggested Citation

  • Omar Al-Ubaydli & John A. List, 2012. "On the Generalizability of Experimental Results in Economics," NBER Working Papers 17957, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:17957
    Note: PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w17957.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bohm, Peter, 1972. "Estimating demand for public goods: An experiment," European Economic Review, Elsevier, vol. 3(2), pages 111-130.
    2. Ernst Fehr & Klaus M. Schmidt, 1999. "A Theory of Fairness, Competition, and Cooperation," The Quarterly Journal of Economics, Oxford University Press, vol. 114(3), pages 817-868.
    3. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-793, September.
    4. Daniel Rondeau & John List, 2008. "Matching and challenge gifts to charity: evidence from laboratory and natural field experiments," Experimental Economics, Springer;Economic Science Association, vol. 11(3), pages 253-267, September.
    5. Kenneth I. Wolpin & Mark R. Rosenzweig, 2000. "Natural "Natural Experiments" in Economics," Journal of Economic Literature, American Economic Association, vol. 38(4), pages 827-874, December.
    6. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    7. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    8. John A. List, 2006. "The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions," Journal of Political Economy, University of Chicago Press, vol. 114(1), pages 1-37, February.
    9. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    10. Glenn W. Harrison & Morten I. Lau & E. Elisabet Rutström, 2007. "Estimating Risk Attitudes in Denmark: A Field Experiment," Scandinavian Journal of Economics, Wiley Blackwell, vol. 109(2), pages 341-368, June.
    11. Vernon L. Smith, 1962. "An Experimental Study of Competitive Market Behavior," Journal of Political Economy, University of Chicago Press, vol. 70, pages 322-322.
    12. John A. List, 2011. "The Market for Charitable Giving," Journal of Economic Perspectives, American Economic Association, vol. 25(2), pages 157-180, Spring.
    13. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," CESifo Working Paper Series 2894, CESifo.
    14. John A. List, 2003. "Does Market Experience Eliminate Market Anomalies?," The Quarterly Journal of Economics, Oxford University Press, vol. 118(1), pages 41-71.
    15. repec:feb:natura:0053 is not listed on IDEAS
    16. John A. List, 2004. "Neoclassical Theory Versus Prospect Theory: Evidence from the Marketplace," Econometrica, Econometric Society, vol. 72(2), pages 615-625, March.
    17. Kahneman, Daniel & Knetsch, Jack L & Thaler, Richard, 1986. "Fairness as a Constraint on Profit Seeking: Entitlements in the Market," American Economic Review, American Economic Association, vol. 76(4), pages 728-741, September.
    18. Muriel Niederle & Carmit Segal & Lise Vesterlund, 2013. "How Costly Is Diversity? Affirmative Action in Light of Gender Differences in Competitiveness," Management Science, INFORMS, vol. 59(1), pages 1-16, May.
    19. repec:feb:artefa:0087 is not listed on IDEAS
    20. John A. List, 2011. "Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 3-16, Summer.
    21. Henrich, Joseph & Boyd, Robert & Bowles, Samuel & Camerer, Colin & Fehr, Ernst & Gintis, Herbert (ed.), 2004. "Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-Scale Societies," OUP Catalogue, Oxford University Press, number 9780199262052.
    22. Peter Bohm, 1972. "Estimating the demand for public goods: An experiment," Framed Field Experiments 00126, The Field Experiments Website.
    23. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    24. Phelps, Edmund S, 1972. "The Statistical Theory of Racism and Sexism," American Economic Review, American Economic Association, vol. 62(4), pages 659-661, September.
    25. repec:feb:artefa:0110 is not listed on IDEAS
    26. James J. Heckman, 2000. "Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, Oxford University Press, vol. 115(1), pages 45-97.
    27. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Francis Larson & John List & Robert Metcalfe, 2016. "Can Myopic Loss Aversion Explain the Equity Premium Puzzle? Evidence from a Natural Field Experiment with Professional Traders," Natural Field Experiments 00534, The Field Experiments Website.
    2. Omar Al-Ubaydli & John List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," Artefactual Field Experiments 00458, The Field Experiments Website.
    3. Beck, Adrian & Kerschbamer, Rudolf & Qiu, Jianying & Sutter, Matthias, 2014. "Car mechanics in the lab––Investigating the behavior of real experts on experimental markets for credence goods," Journal of Economic Behavior & Organization, Elsevier, vol. 108(C), pages 166-173.
    4. Decio Coviello & Andrea Guglielmo & Giancarlo Spagnolo, 2015. "The Effect of Discretion on Procurement Performance," CEIS Research Paper 361, Tor Vergata University, CEIS, revised 17 Nov 2015.
    5. Holmen, Martin & Kirchler, Michael & Kleinlercher, Daniel, 2014. "Do option-like incentives induce overvaluation? Evidence from experimental asset markets," Journal of Economic Dynamics and Control, Elsevier, vol. 40(C), pages 179-194.
    6. Goeschl, Timo & Kettner, Sara Elisa & Lohse, Johannes & Schwieren, Christiane, 2015. "What do we learn from public good games about voluntary climate action? Evidence from an artefactual field experiment," Working Papers 0595, University of Heidelberg, Department of Economics.
    7. Michael L. Anderson & Fangwen Lu, 2017. "Learning to Manage and Managing to Learn: The Effects of Student Leadership Service," Management Science, INFORMS, vol. 63(10), pages 3246-3261, October.
    8. Omar Al-Ubaydli & John A. List, 2013. "On the Generalizability of Experimental Results in Economics: With a Response to Commentors," CESifo Working Paper Series 4543, CESifo.
    9. Matteo M. Galizzi & Daniel Navarro-Martínez, 2015. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Working Papers 802, Barcelona Graduate School of Economics.
    10. Kendel, Adnane & Lazaric, Nathalie & Maréchal, Kevin, 2017. "What do people ‘learn by looking’ at direct feedback on their energy consumption? Results of a field study in Southern France," Energy Policy, Elsevier, vol. 108(C), pages 593-605.
    11. John A. List, 2014. "Using Field Experiments to Change the Template of How We Teach Economics," The Journal of Economic Education, Taylor & Francis Journals, vol. 45(2), pages 81-89, June.
    12. Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2013. "Opting-in: Participation bias in economic experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 43-70.
    13. Michel André Maréchal & Christian Thöni, 2019. "Hidden Persuaders: Do Small Gifts Lubricate Business Negotiations?," Management Science, INFORMS, vol. 65(8), pages 3877-3888, August.
    14. Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2012. "Opting-In: Participation Biases in the Lab," IZA Discussion Papers 6865, Institute of Labor Economics (IZA).
    15. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    16. Egebark, Johan & Ekström, Mathias, 2016. "Can indifference make the world greener?," Journal of Environmental Economics and Management, Elsevier, vol. 76(C), pages 1-13.
    17. Judd Kessler & Judd B. Kessler, 2013. "When will there be Gift Exchange? Addressing the Lab-Field Debate with Laboratory Gift Exchange Experiments," CESifo Working Paper Series 4161, CESifo.
    18. Omar Al-Ubaydli & John A. List & Danielle LoRe & Dana Suskind, 2017. "Scaling for Economists: Lessons from the Non-Adherence Problem in the Medical Literature," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 125-144, Fall.
    19. Rodney J. Andrews & Trevon D. Logan & Michael J. Sinkey, 2012. "Identifying Confirmatory Bias in the Field: Evidence from a Poll of Experts," NBER Working Papers 18064, National Bureau of Economic Research, Inc.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    2. Omar Al-Ubaydli & John A. List, 2013. "On the Generalizability of Experimental Results in Economics: With a Response to Commentors," CESifo Working Paper Series 4543, CESifo.
    3. Omar Al-Ubaydli & John List, 2016. "Field Experiments in Markets," Artefactual Field Experiments j0002, The Field Experiments Website.
    4. List John A., 2007. "Field Experiments: A Bridge between Lab and Naturally Occurring Data," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(2), pages 1-47, April.
    5. Matteo M. Galizzi & Daniel Navarro Martinez, 2015. "On the external validity of social-preference games: A systematic lab-field study," Economics Working Papers 1462, Department of Economics and Business, Universitat Pompeu Fabra.
    6. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    7. Omar Al-Ubaydli & John List, 2019. "How natural field experiments have enhanced our understanding of unemployment," Natural Field Experiments 00649, The Field Experiments Website.
    8. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    9. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    10. Al-Ubaydli, Omar & Boettke, Peter, 2010. "Markets as economizers of information: Field experimental examination of the “Hayek Hypothesis”," MPRA Paper 27660, University Library of Munich, Germany.
    11. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    12. John A. List, 2014. "Using Field Experiments to Change the Template of How We Teach Economics," The Journal of Economic Education, Taylor & Francis Journals, vol. 45(2), pages 81-89, June.
    13. Weimann Joachim, 2015. "Die Rolle von Verhaltensökonomik und experimenteller Forschung in Wirtschaftswissenschaft und Politikberatung," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 16(3), pages 231-252, October.
    14. S. Dellavigna., 2011. "Psychology and Economics: Evidence from the Field," VOPROSY ECONOMIKI, N.P. Redaktsiya zhurnala "Voprosy Economiki", vol. 4.
    15. Omar Al-Ubaydli & John A. List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," NBER Working Papers 20877, National Bureau of Economic Research, Inc.
    16. Eric Floyd & John A. List, 2016. "Using Field Experiments in Accounting and Finance," Journal of Accounting Research, Wiley Blackwell, vol. 54(2), pages 437-475, May.
    17. Gary E. Bolton & Axel Ockenfels, 2008. "Does Laboratory Trading Mirror Behavior in Real World Markets? Fair Bargaining and Competitive Bidding on EBay," CESifo Working Paper Series 2241, CESifo.
    18. John List & Michael Price, 2013. "Using Field Experiments in Environmental and Resource Economics," Artefactual Field Experiments 00447, The Field Experiments Website.
    19. Zhixin Dai & Fabio Galeotti & Marie Claire Villeval, 2018. "Cheating in the Lab Predicts Fraud in the Field: An Experiment in Public Transportation," Management Science, INFORMS, vol. 64(3), pages 1081-1100, March.
    20. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.

    More about this item

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D03 - Microeconomics - - General - - - Behavioral Microeconomics: Underlying Principles

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:17957. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.