IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/15961.html
   My bibliography  Save this paper

The Online Laboratory: Conducting Experiments in a Real Labor Market

Author

Listed:
  • John J. Horton
  • David G. Rand
  • Richard J. Zeckhauser

Abstract

Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid - both internally and externally - as laboratory and field experiments, while requiring far less money and time to design and to conduct. In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results. We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices. We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves. In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats. We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question. We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.

Suggested Citation

  • John J. Horton & David G. Rand & Richard J. Zeckhauser, 2010. "The Online Laboratory: Conducting Experiments in a Real Labor Market," NBER Working Papers 15961, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:15961
    Note: LE LS
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w15961.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    2. Tyran, Jean-Robert, 2003. "Behavioral Game Theory. Experiments in Strategic Interaction: Colin F. Camerer, Princeton University Press, Princeton, New Jersey, 2003, p. 550, Price $65.00/[UK pound]42.95, ISBN 0-691-09039-4," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 32(6), pages 717-720, December.
    3. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    4. Daniel J. Benjamin & James J. Choi & Geoffrey Fisher, 2016. "Religious Identity and Economic Behavior," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 617-637, October.
    5. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2010. "Social Incentives in the Workplace," Review of Economic Studies, Oxford University Press, vol. 77(2), pages 417-458.
    6. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    7. Ernst Fehr & Klaus M. Schmidt, 1999. "A Theory of Fairness, Competition, and Cooperation," The Quarterly Journal of Economics, Oxford University Press, vol. 114(3), pages 817-868.
    8. Alexandre Mas & Enrico Moretti, 2009. "Peers at Work," American Economic Review, American Economic Association, vol. 99(1), pages 112-145, March.
    9. Catherine Eckel & Rick Wilson, 2006. "Internet cautions: Experimental games with internet partners," Experimental Economics, Springer;Economic Science Association, vol. 9(1), pages 53-66, April.
    10. Iris Bohnet & Fiona Greig & Benedikt Herrmann & Richard Zeckhauser, 2008. "Betrayal Aversion: Evidence from Brazil, China, Oman, Switzerland, Turkey, and the United States," American Economic Review, American Economic Association, vol. 98(1), pages 294-310, March.
    11. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    12. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2009. "Social Connections and Incentives in the Workplace: Evidence From Personnel Data," Econometrica, Econometric Society, vol. 77(4), pages 1047-1094, July.
    13. Daniel J. Benjamin & James J. Choi & A. Joshua Strickland, 2010. "Social Identity and Preferences," American Economic Review, American Economic Association, vol. 100(4), pages 1913-1928, September.
    14. Jordi Brandts & Gary Charness, 2000. "Hot vs. Cold: Sequential Responses and Preference Stability in Experimental Games," Experimental Economics, Springer;Economic Science Association, vol. 2(3), pages 227-238, March.
    15. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    16. Siddharth Suri & Duncan J Watts, 2011. "Cooperation and Contagion in Web-Based, Networked Public Goods Experiments," PLOS ONE, Public Library of Science, vol. 6(3), pages 1-18, March.
    17. Horton, John J., 2011. "The condition of the Turking class: Are online employers fair and honest?," Economics Letters, Elsevier, vol. 111(1), pages 10-12, April.
    18. Uri Gneezy & Kenneth L. Leonard & John A. List, 2009. "Gender Differences in Competition: Evidence From a Matrilineal and a Patriarchal Society," Econometrica, Econometric Society, vol. 77(5), pages 1637-1664, September.
    19. Paul Resnick & Richard Zeckhauser & John Swanson & Kate Lockwood, 2006. "The value of reputation on eBay: A controlled experiment," Experimental Economics, Springer;Economic Science Association, vol. 9(2), pages 79-101, June.
    20. Martin G. Kocher & Matthias Sutter, 2005. "The Decision Maker Matters: Individual Versus Group Behaviour in Experimental Beauty-Contest Games," Economic Journal, Royal Economic Society, vol. 115(500), pages 200-223, January.
    21. Andreoni, James, 1990. "Impure Altruism and Donations to Public Goods: A Theory of Warm-Glow Giving?," Economic Journal, Royal Economic Society, vol. 100(401), pages 464-477, June.
    22. Benedikt Herrmann & Christian Thöni, 2009. "Measuring conditional cooperation: a replication study in Russia," Experimental Economics, Springer;Economic Science Association, vol. 12(1), pages 87-92, March.
    23. Simon Gachter & Ernst Fehr, 2000. "Cooperation and Punishment in Public Goods Experiments," American Economic Review, American Economic Association, vol. 90(4), pages 980-994, September.
    24. repec:feb:artefa:0087 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jiang, Jiang & Li, Sherry Xin, 2019. "Group identity and partnership," Journal of Economic Behavior & Organization, Elsevier, vol. 160(C), pages 202-213.
    2. Paetzel, Fabian & Sausgruber, Rupert, 2018. "Cognitive ability and in-group bias: An experimental study," Journal of Public Economics, Elsevier, vol. 167(C), pages 280-292.
    3. An, Jiafu & Jiang, Mengfei & Xu, Jiaman, 2021. "Professional norms and risk-taking of bank employees: Do expectations of peers’ risk preferences matter?," Journal of Financial Stability, Elsevier, vol. 56(C).
    4. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    5. Anya Savikhin Samek & Roman Sheremeta, 2014. "Recognizing contributors: an experiment on public goods," Experimental Economics, Springer;Economic Science Association, vol. 17(4), pages 673-690, December.
    6. John J. Horton & Richard J. Zeckhauser, 2016. "The Causes of Peer Effects in Production: Evidence from a Series of Field Experiments," NBER Working Papers 22386, National Bureau of Economic Research, Inc.
    7. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    8. Alain Cohn & Ernst Fehr & Lorenz Goette, 2015. "Fair Wages and Effort Provision: Combining Evidence from a Choice Experiment and a Field Experiment," Management Science, INFORMS, vol. 61(8), pages 1777-1794, August.
    9. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2011. "Field Experiments with Firms," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 63-82, Summer.
    10. Robert Oxoby, 2013. "Paretian dictators: constraining choice in a voluntary contribution game," Constitutional Political Economy, Springer, vol. 24(2), pages 125-138, June.
    11. Lammers, Frauke & Schiller, Jörg, 2010. "Contract design and insurance fraud: An experimental investigation," FZID Discussion Papers 19-2010, University of Hohenheim, Center for Research on Innovation and Services (FZID).
    12. Goto, Jun & Sawada, Yasuyuki & Aida, Takeshi & Aoyagi, Keitaro, 2015. "Incentives and Social Preferences: Experimental Evidence from a Seemingly Inefficienct Traditional Labor Contract," 2015 Conference, August 9-14, 2015, Milan, Italy 211687, International Association of Agricultural Economists.
    13. Simon Gaechter, 2014. "Human Pro-Social Motivation and the Maintenance of Social Order," CESifo Working Paper Series 4729, CESifo.
    14. Ananish Chaudhuri, 2011. "Sustaining cooperation in laboratory public goods experiments: a selective survey of the literature," Experimental Economics, Springer;Economic Science Association, vol. 14(1), pages 47-83, March.
    15. Cooper, David J. & Saral, Krista & Villeval, Marie Claire, 2019. "Why Join a Team?," IZA Discussion Papers 12587, Institute of Labor Economics (IZA).
      • David J. Cooper & Krista Saral & Marie Claire Villeval, 2021. "Why Join a Team?," Post-Print halshs-03003653, HAL.
      • David Cooper & Krista Saral & Marie Claire Villeval, 2019. "Why Join a Team?," Working Papers halshs-02295921, HAL.
      • David J. Cooper & Krista Saral & Marie Claire Villeval, 2019. "Why Join a Team?," Working Papers 1928, Groupe d'Analyse et de Théorie Economique Lyon St-Étienne (GATE Lyon St-Étienne), Université de Lyon.
    16. Yamada, Katsunori & Sato, Masayuki, 2013. "Another avenue for anatomy of income comparisons: Evidence from hypothetical choice experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 89(C), pages 35-57.
    17. Amrei Lahno & Marta Serra-Garcia, 2015. "Peer effects in risk taking: Envy or conformity?," Journal of Risk and Uncertainty, Springer, vol. 50(1), pages 73-95, February.
    18. Julie Beugnot & Bernard Fortin & Guy Lacroix & Marie Claire Villeval, 2013. "Social Networks and Peer Effects at Work," Cahiers de recherche 1320, CIRPEE.

    More about this item

    JEL classification:

    • C70 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - General
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • J2 - Labor and Demographic Economics - - Demand and Supply of Labor

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:15961. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.