IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/20877.html
   My bibliography  Save this paper

Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model

Author

Listed:
  • Omar Al-Ubaydli
  • John A. List

Abstract

A commonly held view is that laboratory experiments provide researchers with more “control” than natural field experiments, and that this advantage is to be balanced against the disadvantage that laboratory experiments are less generalizable. This paper presents a simple model that explores circumstances under which natural field experiments provide researchers with more control than laboratory experiments afford. This stems from the covertness of natural field experiments: laboratory experiments provide researchers with a high degree of control in the environment which participants agree to be experimental subjects. When participants systematically opt out of laboratory experiments, the researcher’s ability to manipulate certain variables is limited. In contrast, natural field experiments bypass the participation decision altogether and allow for a potentially more diverse participant pool within the market of interest. We show one particular case where such selection is invaluable: when treatment effects interact with participant characteristics.

Suggested Citation

  • Omar Al-Ubaydli & John A. List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," NBER Working Papers 20877, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:20877
    Note: EEE PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w20877.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Colin F. Camerer, 1998. "Can Asset Markets Be Manipulated? A Field Experiment with Racetrack Betting," Journal of Political Economy, University of Chicago Press, vol. 106(3), pages 457-482, June.
    2. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    3. repec:feb:framed:0081 is not listed on IDEAS
    4. Castillo, Marco & Petrie, Ragan & Torero, Maximo & Vesterlund, Lise, 2013. "Gender differences in bargaining outcomes: A field experiment on discrimination," Journal of Public Economics, Elsevier, vol. 99(C), pages 35-48.
    5. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    6. Uri Gneezy & John List & Michael K. Price, 2012. "Toward an Understanding of Why People Discriminate: Evidence from a Series of Natural Field Experiments," NBER Working Papers 17855, National Bureau of Economic Research, Inc.
    7. Glenn W. Harrison & Morten I. Lau & E. Elisabet Rutström, 2007. "Estimating Risk Attitudes in Denmark: A Field Experiment," Scandinavian Journal of Economics, Wiley Blackwell, vol. 109(2), pages 341-368, June.
    8. Omar Al-Ubaydli & John List, 2012. "On the Generalizability of Experimental Results in Economics," Artefactual Field Experiments 00467, The Field Experiments Website.
    9. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    10. Alan S. Gerber & Dean Karlan & Daniel Bergan, 2009. "Does the Media Matter? A Field Experiment Measuring the Effect of Newspapers on Voting Behavior and Political Opinions," American Economic Journal: Applied Economics, American Economic Association, vol. 1(2), pages 35-52, April.
    11. Blair Cleave & Nikos Nikiforakis & Robert Slonim, 2013. "Is there selection bias in laboratory experiments? The case of social and risk preferences," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 372-382, September.
    12. Harrison, Glenn W. & Lau, Morten I. & Elisabet Rutström, E., 2009. "Risk attitudes, randomization to treatment, and self-selection into experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 70(3), pages 498-507, June.
    13. Armin Falk, 2007. "Gift Exchange in the Field," Econometrica, Econometric Society, vol. 75(5), pages 1501-1511, September.
    14. John A. List, 2004. "Testing Neoclassical Competitive Theory in Multilateral Decentralized Markets," Journal of Political Economy, University of Chicago Press, vol. 112(5), pages 1131-1156, October.
    15. Marianne Bertrand & Sendhil Mullainathan, 2004. "Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination," American Economic Review, American Economic Association, vol. 94(4), pages 991-1013, September.
    16. Neal S Young, 2008. "Why Current Publication May Distort Science," Working Papers id:1757, eSocialSciences.
    17. Neal S Young & John P A Ioannidis & Omar Al-Ubaydli, 2008. "Why Current Publication Practices May Distort Science," PLOS Medicine, Public Library of Science, vol. 5(10), pages 1-5, October.
    18. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    19. Smith, Vernon L & Walker, James M, 1993. "Monetary Rewards and Decision Cost in Experimental Economics," Economic Inquiry, Western Economic Association International, vol. 31(2), pages 245-261, April.
    20. John A. List, 2003. "Does Market Experience Eliminate Market Anomalies?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(1), pages 41-71.
    21. Colin Camerer, 1998. "Can asset markets be manipulated? A field experiment with racetrack betting," Natural Field Experiments 00222, The Field Experiments Website.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ubaydli & John List, 2015. "Control in Experiments: A Simple Model," Artefactual Field Experiments 00397, The Field Experiments Website.
    2. Omar Al-Ubaydli & John List, 2016. "Field Experiments in Markets," Artefactual Field Experiments j0002, The Field Experiments Website.
    3. Omar Al-Ubaydli & John A. List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control Than Laboratory Experiments?," American Economic Review, American Economic Association, vol. 105(5), pages 462-466, May.
    4. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    5. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    6. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    7. Omar Al-Ubaydli & John A. List, 2013. "On the Generalizability of Experimental Results in Economics: With a Response to Commentors," CESifo Working Paper Series 4543, CESifo.
    8. John A. List, 2014. "Using Field Experiments to Change the Template of How We Teach Economics," The Journal of Economic Education, Taylor & Francis Journals, vol. 45(2), pages 81-89, June.
    9. Brit Grosskopf & Graeme Pearce, 2020. "Do You Mind Me Paying Less? Measuring Other-Regarding Preferences in the Market for Taxis," Management Science, INFORMS, vol. 66(11), pages 5059-5074, November.
    10. David Neumark, 2018. "Experimental Research on Labor Market Discrimination," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 799-866, September.
    11. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    12. Steven D. Levitt & John A. List, 2007. "Viewpoint: On the generalizability of lab behaviour to the field," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(2), pages 347-370, May.
    13. James Alm & William D. Schulze & Carrie von Bose & Jubo Yan, 2019. "Appeals to Social Norms and Taxpayer Compliance," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 638-666, October.
    14. Jonathan E. Alevy & Craig E. Landry & John A. List, 2015. "Field Experiments On The Anchoring Of Economic Valuations," Economic Inquiry, Western Economic Association International, vol. 53(3), pages 1522-1538, July.
    15. Michel André Maréchal & Christian Thöni, 2019. "Hidden Persuaders: Do Small Gifts Lubricate Business Negotiations?," Management Science, INFORMS, vol. 65(8), pages 3877-3888, August.
    16. Frijters, Paul & Kong, Tao Sherry & Liu, Elaine M., 2015. "Who is coming to the artefactual field experiment? Participation bias among Chinese rural migrants," Journal of Economic Behavior & Organization, Elsevier, vol. 114(C), pages 62-74.
    17. Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2013. "Opting-in: Participation bias in economic experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 43-70.
    18. Omar Al-Ubaydli & John A. List, 2012. "On the Generalizability of Experimental Results in Economics," NBER Working Papers 17957, National Bureau of Economic Research, Inc.
    19. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    20. List John A., 2007. "Field Experiments: A Bridge between Lab and Naturally Occurring Data," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 6(2), pages 1-47, April.

    More about this item

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:20877. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.