IDEAS home Printed from https://ideas.repec.org/a/kap/expeco/v14y2011i3p399-425.html
   My bibliography  Save this article

The online laboratory: conducting experiments in a real labor market

Author

Listed:
  • John Horton
  • David Rand
  • Richard Zeckhauser

Abstract

Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid – both internally and externally – as laboratory and field experiments, while requiring far less money and time to design and to conduct. In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results. We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices. We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves. In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats. We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question. We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
  • Handle: RePEc:kap:expeco:v:14:y:2011:i:3:p:399-425
    DOI: 10.1007/s10683-011-9273-9
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s10683-011-9273-9
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1007/s10683-011-9273-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Horton, John J., 2011. "The condition of the Turking class: Are online employers fair and honest?," Economics Letters, Elsevier, vol. 111(1), pages 10-12, April.
    2. Ernst Fehr & Klaus M. Schmidt, 1999. "A Theory of Fairness, Competition, and Cooperation," The Quarterly Journal of Economics, Oxford University Press, vol. 114(3), pages 817-868.
    3. Alexandre Mas & Enrico Moretti, 2009. "Peers at Work," American Economic Review, American Economic Association, vol. 99(1), pages 112-145, March.
    4. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    5. Uri Gneezy & Kenneth L. Leonard & John A. List, 2009. "Gender Differences in Competition: Evidence From a Matrilineal and a Patriarchal Society," Econometrica, Econometric Society, vol. 77(5), pages 1637-1664, September.
    6. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    7. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2009. "Social Connections and Incentives in the Workplace: Evidence From Personnel Data," Econometrica, Econometric Society, vol. 77(4), pages 1047-1094, July.
    8. Tyran, Jean-Robert, 2003. "Behavioral Game Theory. Experiments in Strategic Interaction: Colin F. Camerer, Princeton University Press, Princeton, New Jersey, 2003, p. 550, Price $65.00/[UK pound]42.95, ISBN 0-691-09039-4," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 32(6), pages 717-720, December.
    9. Paul Resnick & Richard Zeckhauser & John Swanson & Kate Lockwood, 2006. "The value of reputation on eBay: A controlled experiment," Experimental Economics, Springer;Economic Science Association, vol. 9(2), pages 79-101, June.
    10. Jordi Brandts & Gary Charness, 2000. "Hot vs. Cold: Sequential Responses and Preference Stability in Experimental Games," Experimental Economics, Springer;Economic Science Association, vol. 2(3), pages 227-238, March.
    11. Martin G. Kocher & Matthias Sutter, 2005. "The Decision Maker Matters: Individual Versus Group Behaviour in Experimental Beauty-Contest Games," Economic Journal, Royal Economic Society, vol. 115(500), pages 200-223, January.
    12. Daniel J. Benjamin & James J. Choi & Geoffrey Fisher, 2016. "Religious Identity and Economic Behavior," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 617-637, October.
    13. Andreoni, James, 1990. "Impure Altruism and Donations to Public Goods: A Theory of Warm-Glow Giving?," Economic Journal, Royal Economic Society, vol. 100(401), pages 464-477, June.
    14. Benedikt Herrmann & Christian Thöni, 2009. "Measuring conditional cooperation: a replication study in Russia," Experimental Economics, Springer;Economic Science Association, vol. 12(1), pages 87-92, March.
    15. Catherine Eckel & Rick Wilson, 2006. "Internet cautions: Experimental games with internet partners," Experimental Economics, Springer;Economic Science Association, vol. 9(1), pages 53-66, April.
    16. Iris Bohnet & Fiona Greig & Benedikt Herrmann & Richard Zeckhauser, 2008. "Betrayal Aversion: Evidence from Brazil, China, Oman, Switzerland, Turkey, and the United States," American Economic Review, American Economic Association, vol. 98(1), pages 294-310, March.
    17. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    18. Simon Gachter & Ernst Fehr, 2000. "Cooperation and Punishment in Public Goods Experiments," American Economic Review, American Economic Association, vol. 90(4), pages 980-994, September.
    19. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dimant, Eugen, 2015. "On Peer Effects: Behavioral Contagion of (Un)Ethical Behavior and the Role of Social Identity," MPRA Paper 68732, University Library of Munich, Germany.
    2. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    3. Anya Savikhin Samek & Roman Sheremeta, 2014. "Recognizing contributors: an experiment on public goods," Experimental Economics, Springer;Economic Science Association, vol. 17(4), pages 673-690, December.
    4. Robert Oxoby, 2013. "Paretian dictators: constraining choice in a voluntary contribution game," Constitutional Political Economy, Springer, vol. 24(2), pages 125-138, June.
    5. Lammers, Frauke & Schiller, Jörg, 2010. "Contract design and insurance fraud: An experimental investigation," FZID Discussion Papers 19-2010, University of Hohenheim, Center for Research on Innovation and Services (FZID).
    6. Simon Gaechter, 2014. "Human Pro-Social Motivation and the Maintenance of Social Order," CESifo Working Paper Series 4729, CESifo.
    7. Amrei Lahno & Marta Serra-Garcia, 2015. "Peer effects in risk taking: Envy or conformity?," Journal of Risk and Uncertainty, Springer, vol. 50(1), pages 73-95, February.
    8. Barmettler, Franziska & Fehr, Ernst & Zehnder, Christian, 2012. "Big experimenter is watching you! Anonymity and prosocial behavior in the laboratory," Games and Economic Behavior, Elsevier, vol. 75(1), pages 17-34.
    9. Omar Al-Ubaydli & John List, 2016. "Field Experiments in Markets," Artefactual Field Experiments j0002, The Field Experiments Website.
    10. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    11. Martin G. Kocher & Peter Martinsson & Kristian Ove R. Myrseth & Conny E. Wollbrant, 2017. "Strong, bold, and kind: self-control and cooperation in social dilemmas," Experimental Economics, Springer;Economic Science Association, vol. 20(1), pages 44-69, March.
    12. Bodo Sturm & Joachim Weimann, 2006. "Experiments in Environmental Economics and Some Close Relatives," Journal of Economic Surveys, Wiley Blackwell, vol. 20(3), pages 419-457, July.
    13. Stephen Cheung, 2014. "New insights into conditional cooperation and punishment from a strategy method experiment," Experimental Economics, Springer;Economic Science Association, vol. 17(1), pages 129-153, March.
    14. Jon Anderson & Stephen Burks & Jeffrey Carpenter & Lorenz Götte & Karsten Maurer & Daniele Nosenzo & Ruth Potter & Kim Rocha & Aldo Rustichini, 2013. "Self-selection and variations in the laboratory measurement of other-regarding preferences across subject pools: evidence from one college student and two adult samples," Experimental Economics, Springer;Economic Science Association, vol. 16(2), pages 170-189, June.
    15. Nagore Iriberri & Pedro Rey-Biel, 2011. "The role of role uncertainty in modified dictator games," Experimental Economics, Springer;Economic Science Association, vol. 14(2), pages 160-180, May.
    16. Nyborg, Karine, 2018. "Reciprocal climate negotiators," Journal of Environmental Economics and Management, Elsevier, vol. 92(C), pages 707-725.
    17. Arno Riedl & Ingrid M. T. Rohde & Martin Strobel, 2016. "Efficient Coordination in Weakest-Link Games," Review of Economic Studies, Oxford University Press, vol. 83(2), pages 737-767.
    18. Christian Thoeni & Simon Gaechter, 2011. "Peer Effects and Social Preferences in Voluntary Cooperation," Discussion Papers 2011-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    19. Cárdenas, Juan Camilo & Chong, Alberto & Ñopo, Hugo, 2013. "Stated social behavior and revealed actions: Evidence from six Latin American countries," Journal of Development Economics, Elsevier, vol. 104(C), pages 16-33.
    20. Bogliacino, Francesco & Codagnone, Cristiano, 2021. "Microfoundations, behaviour, and evolution: Evidence from experiments," Structural Change and Economic Dynamics, Elsevier, vol. 56(C), pages 372-385.

    More about this item

    Keywords

    Experimentation; Online labor markets; Prisoner’s dilemma; Field experiment; Internet; J2; C93; C91; C92; C70;
    All these keywords.

    JEL classification:

    • C70 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - General
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • J2 - Labor and Demographic Economics - - Demand and Supply of Labor

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:expeco:v:14:y:2011:i:3:p:399-425. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.springer.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.