IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/15961.html
   My bibliography  Save this paper

The Online Laboratory: Conducting Experiments in a Real Labor Market

Author

Listed:
  • John J. Horton
  • David G. Rand
  • Richard J. Zeckhauser

Abstract

Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid - both internally and externally - as laboratory and field experiments, while requiring far less money and time to design and to conduct. In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results. We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices. We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves. In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats. We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question. We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.

Suggested Citation

  • John J. Horton & David G. Rand & Richard J. Zeckhauser, 2010. "The Online Laboratory: Conducting Experiments in a Real Labor Market," NBER Working Papers 15961, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:15961
    Note: LE LS
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w15961.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    2. Tyran, Jean-Robert, 2003. "Behavioral Game Theory. Experiments in Strategic Interaction: Colin F. Camerer, Princeton University Press, Princeton, New Jersey, 2003, p. 550, Price $65.00/[UK pound]42.95, ISBN 0-691-09039-4," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 32(6), pages 717-720, December.
    3. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    4. Daniel J. Benjamin & James J. Choi & Geoffrey Fisher, 2016. "Religious Identity and Economic Behavior," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 617-637, October.
    5. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2010. "Social Incentives in the Workplace," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 77(2), pages 417-458.
    6. Ernst Fehr & Klaus M. Schmidt, 1999. "A Theory of Fairness, Competition, and Cooperation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(3), pages 817-868.
    7. Alexandre Mas & Enrico Moretti, 2009. "Peers at Work," American Economic Review, American Economic Association, vol. 99(1), pages 112-145, March.
    8. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    9. Catherine Eckel & Rick Wilson, 2006. "Internet cautions: Experimental games with internet partners," Experimental Economics, Springer;Economic Science Association, vol. 9(1), pages 53-66, April.
    10. Iris Bohnet & Fiona Greig & Benedikt Herrmann & Richard Zeckhauser, 2008. "Betrayal Aversion: Evidence from Brazil, China, Oman, Switzerland, Turkey, and the United States," American Economic Review, American Economic Association, vol. 98(1), pages 294-310, March.
    11. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    12. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2009. "Social Connections and Incentives in the Workplace: Evidence From Personnel Data," Econometrica, Econometric Society, vol. 77(4), pages 1047-1094, July.
    13. Daniel J. Benjamin & James J. Choi & A. Joshua Strickland, 2010. "Social Identity and Preferences," American Economic Review, American Economic Association, vol. 100(4), pages 1913-1928, September.
    14. Jordi Brandts & Gary Charness, 2000. "Hot vs. Cold: Sequential Responses and Preference Stability in Experimental Games," Experimental Economics, Springer;Economic Science Association, vol. 2(3), pages 227-238, March.
    15. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    16. Siddharth Suri & Duncan J Watts, 2011. "Cooperation and Contagion in Web-Based, Networked Public Goods Experiments," PLOS ONE, Public Library of Science, vol. 6(3), pages 1-18, March.
    17. Horton, John J., 2011. "The condition of the Turking class: Are online employers fair and honest?," Economics Letters, Elsevier, vol. 111(1), pages 10-12, April.
    18. Uri Gneezy & Kenneth L. Leonard & John A. List, 2009. "Gender Differences in Competition: Evidence From a Matrilineal and a Patriarchal Society," Econometrica, Econometric Society, vol. 77(5), pages 1637-1664, September.
    19. Paul Resnick & Richard Zeckhauser & John Swanson & Kate Lockwood, 2006. "The value of reputation on eBay: A controlled experiment," Experimental Economics, Springer;Economic Science Association, vol. 9(2), pages 79-101, June.
    20. Martin G. Kocher & Matthias Sutter, 2005. "The Decision Maker Matters: Individual Versus Group Behaviour in Experimental Beauty-Contest Games," Economic Journal, Royal Economic Society, vol. 115(500), pages 200-223, January.
    21. Andreoni, James, 1990. "Impure Altruism and Donations to Public Goods: A Theory of Warm-Glow Giving?," Economic Journal, Royal Economic Society, vol. 100(401), pages 464-477, June.
    22. Benedikt Herrmann & Christian Thöni, 2009. "Measuring conditional cooperation: a replication study in Russia," Experimental Economics, Springer;Economic Science Association, vol. 12(1), pages 87-92, March.
    23. Simon Gachter & Ernst Fehr, 2000. "Cooperation and Punishment in Public Goods Experiments," American Economic Review, American Economic Association, vol. 90(4), pages 980-994, September.
    24. repec:feb:artefa:0087 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dimant, Eugen, 2015. "On Peer Effects: Behavioral Contagion of (Un)Ethical Behavior and the Role of Social Identity," MPRA Paper 68732, University Library of Munich, Germany.
    2. Jiang, Jiang & Li, Sherry Xin, 2019. "Group identity and partnership," Journal of Economic Behavior & Organization, Elsevier, vol. 160(C), pages 202-213.
    3. Christian Thoeni & Simon Gaechter, 2011. "Peer Effects and Social Preferences in Voluntary Cooperation," Discussion Papers 2011-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    4. Anya Savikhin Samek & Roman Sheremeta, 2014. "Recognizing contributors: an experiment on public goods," Experimental Economics, Springer;Economic Science Association, vol. 17(4), pages 673-690, December.
    5. Horton, John J. & Zeckhauser, Richard J., 2016. "The Causes of Peer Effects in Production: Evidence from a Series of Field Experiments," Working Paper Series 16-027, Harvard University, John F. Kennedy School of Government.
    6. Robert Oxoby, 2013. "Paretian dictators: constraining choice in a voluntary contribution game," Constitutional Political Economy, Springer, vol. 24(2), pages 125-138, June.
    7. Lammers, Frauke & Schiller, Jörg, 2010. "Contract design and insurance fraud: An experimental investigation," FZID Discussion Papers 19-2010, University of Hohenheim, Center for Research on Innovation and Services (FZID).
    8. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    9. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2011. "Field Experiments with Firms," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 63-82, Summer.
    10. Goto, Jun & Sawada, Yasuyuki & Aida, Takeshi & Aoyagi, Keitaro, 2015. "Incentives and Social Preferences: Experimental Evidence from a Seemingly Inefficienct Traditional Labor Contract," 2015 Conference, August 9-14, 2015, Milan, Italy 211687, International Association of Agricultural Economists.
    11. Simon Gaechter, 2014. "Human Pro-Social Motivation and the Maintenance of Social Order," Discussion Papers 2014-02, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    12. Rod Falvey & Tom Lane & Shravan Luckraz, 2022. "On a mechanism that improves efficiency and reduces inequality in voluntary contribution games," Discussion Papers 2022-15, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    13. Paetzel, Fabian & Sausgruber, Rupert, 2018. "Cognitive ability and in-group bias: An experimental study," Journal of Public Economics, Elsevier, vol. 167(C), pages 280-292.
    14. An, Jiafu & Jiang, Mengfei & Xu, Jiaman, 2021. "Professional norms and risk-taking of bank employees: Do expectations of peers’ risk preferences matter?," Journal of Financial Stability, Elsevier, vol. 56(C).
    15. Alain Cohn & Ernst Fehr & Lorenz Goette, 2015. "Fair Wages and Effort Provision: Combining Evidence from a Choice Experiment and a Field Experiment," Management Science, INFORMS, vol. 61(8), pages 1777-1794, August.
    16. Ananish Chaudhuri, 2011. "Sustaining cooperation in laboratory public goods experiments: a selective survey of the literature," Experimental Economics, Springer;Economic Science Association, vol. 14(1), pages 47-83, March.
    17. David J. Cooper & Krista Saral & Marie Claire Villeval, 2021. "Why Join a Team?," Management Science, INFORMS, vol. 67(11), pages 6980-6997, November.
    18. Amrei Lahno & Marta Serra-Garcia, 2015. "Peer effects in risk taking: Envy or conformity?," Journal of Risk and Uncertainty, Springer, vol. 50(1), pages 73-95, February.
    19. Julie Beugnot & Bernard Fortin & Guy Lacroix & Marie Claire Villeval, 2013. "Social Networks and Peer Effects at Work," Cahiers de recherche 1320, CIRPEE.
    20. Matthias Stefan & Jürgen Huber & Michael Kirchler & Matthias Sutter & Markus Walzl, 2020. "Monetary and Social Incentives in Multi-Tasking: The Ranking Substitution Effect," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2020_10, Max Planck Institute for Research on Collective Goods.

    More about this item

    JEL classification:

    • C70 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - General
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • J2 - Labor and Demographic Economics - - Demand and Supply of Labor

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:15961. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.