IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp10517.html
   My bibliography  Save this paper

Conducting Interactive Experiments Online

Author

Listed:
  • Arechar, Antonio A.

    (Yale University)

  • Gächter, Simon

    (University of Nottingham)

  • Molleman, Lucas

    (Max Planck Institute for Human Development)

Abstract

Online labor markets provide new opportunities for behavioral research, but conducting economic experiments online raises important methodological challenges. This particularly holds for interactive designs. In this paper, we provide a methodological discussion of the similarities and differences between interactive experiments conducted in the laboratory and online. To this end, we conduct a repeated public goods experiment with and without punishment using samples from the laboratory and the online platform Amazon Mechanical Turk. We chose to replicate this experiment because it is long and logistically complex. It therefore provides a good case study for discussing the methodological and practical challenges of online interactive experimentation. We find that basic behavioral patterns of cooperation and punishment in the laboratory are replicable online. The most important challenge of online interactive experiments is participant dropout. We discuss measures for reducing dropout and show that, for our case study, dropouts are exogenous to the experiment. We conclude that data quality for interactive experiments via the Internet is adequate and reliable, making online interactive experimentation a valuable complement to laboratory studies.

Suggested Citation

  • Arechar, Antonio A. & Gächter, Simon & Molleman, Lucas, 2017. "Conducting Interactive Experiments Online," IZA Discussion Papers 10517, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp10517
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp10517.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jérôme Hergueux & Nicolas Jacquemet, 2015. "Social preferences in the online laboratory: a randomized experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 251-283, June.
    2. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    3. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    4. repec:hal:pseose:halshs-00984211 is not listed on IDEAS
    5. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, "undated". "The Average Laboratory Samples a Population of 7,300 Amazon Mechanical Turk Workers," Mathematica Policy Research Reports f97b669c7b3e4c2ab95c9f805, Mathematica Policy Research.
    6. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    7. Ben Greiner, 2015. "Subject pool recruitment procedures: organizing experiments with ORSEE," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 1(1), pages 114-125, July.
    8. Jenkins, Stephen P, 1995. "Easy Estimation Methods for Discrete-Time Duration Models," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 57(1), pages 129-138, February.
    9. Katrin Schmelz & Anthony Ziegelmeyer, 2015. "Social Distance and Control Aversion: Evidence from the Internet and the Laboratory," TWI Research Paper Series 100, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    10. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    11. Anderhub, Vital & Muller, Rudolf & Schmidt, Carsten, 2001. "Design and evaluation of an economic experiment via the Internet," Journal of Economic Behavior & Organization, Elsevier, vol. 46(2), pages 227-247, October.
    12. Jon Anderson & Stephen Burks & Jeffrey Carpenter & Lorenz Götte & Karsten Maurer & Daniele Nosenzo & Ruth Potter & Kim Rocha & Aldo Rustichini, 2013. "Self-selection and variations in the laboratory measurement of other-regarding preferences across subject pools: evidence from one college student and two adult samples," Experimental Economics, Springer;Economic Science Association, vol. 16(2), pages 170-189, June.
    13. Johannes Abeler & Daniele Nosenzo, 2015. "Self-selection into laboratory experiments: pro-social motives versus monetary incentives," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 195-214, June.
    14. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, 2015. "The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(5), pages 479-491, September.
    15. Gächter, Simon & Herrmann, Benedikt, 2011. "The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia," European Economic Review, Elsevier, vol. 55(2), pages 193-210, February.
    16. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    17. Jan Stoop & Charles N. Noussair & Daan van Soest, 2012. "From the Lab to the Field: Cooperation among Fishermen," Journal of Political Economy, University of Chicago Press, vol. 120(6), pages 1027-1056.
    18. Krupnikov, Yanna & Levine, Adam Seth, 2014. "Cross-Sample Comparisons and External Validity," Journal of Experimental Political Science, Cambridge University Press, vol. 1(1), pages 59-80, April.
    19. Blair Cleave & Nikos Nikiforakis & Robert Slonim, 2013. "Is there selection bias in laboratory experiments? The case of social and risk preferences," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 372-382, September.
    20. Jesse Chandler & Gabriele Paolacci & Eyal Peer & Pam Mueller & Kate A. Ratliff, 2015. "Using Nonnaive Participants Can Reduce Effect Sizes," Mathematica Policy Research Reports bffac982a56e4cfba3659e74a, Mathematica Policy Research.
    21. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    22. Chesney, Thomas & Chuah, Swee-Hoon & Hoffmann, Robert, 2009. "Virtual world experimentation: An exploratory study," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 618-635, October.
    23. Ernst Fehr & Simon Gächter, 2002. "Altruistic punishment in humans," Nature, Nature, vol. 415(6868), pages 137-140, January.
    24. Jeffrey Carpenter & Erika Seki, 2011. "Do Social Preferences Increase Productivity? Field Experimental Evidence From Fishermen In Toyama Bay," Economic Inquiry, Western Economic Association International, vol. 49(2), pages 612-630, April.
    25. Michael Birnbaum, 2000. "Psychological experiments on the internet," Framed Field Experiments 00125, The Field Experiments Website.
    26. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    27. Gachter, Simon & Herrmann, Benedikt & Thoni, Christian, 2004. "Trust, voluntary cooperation, and socio-economic background: survey and experimental evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 55(4), pages 505-531, December.
    28. Simon Gachter & Ernst Fehr, 2000. "Cooperation and Punishment in Public Goods Experiments," American Economic Review, American Economic Association, vol. 90(4), pages 980-994, September.
    29. Ofra Amir & David G Rand & Ya'akov Kobi Gal, 2012. "Economic Games on the Internet: The Effect of $1 Stakes," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-4, February.
    30. Michal Krawczyk, 2011. "What brings your subjects to the lab? A field experiment," Natural Field Experiments 00694, The Field Experiments Website.
    31. John A. List, 2004. "Young, Selfish and Male: Field evidence of social preferences," Economic Journal, Royal Economic Society, vol. 114(492), pages 121-149, January.
    32. Michèle Belot & Raymond Duch & Luis Miller, 2010. "Who should be called to the lab? A comprehensive comparison of students and non-students in classic experimental games," Discussion Papers 2010001, University of Oxford, Nuffield College.
    33. David G. Rand & Alexander Peysakhovich & Gordon T. Kraft-Todd & George E. Newman & Owen Wurzbacher & Martin A. Nowak & Joshua D. Greene, 2014. "Social heuristics shape intuitive cooperation," Nature Communications, Nature, vol. 5(1), pages 1-12, May.
    34. Bock, Olaf & Baetge, Ingmar & Nicklisch, Andreas, 2014. "hroot: Hamburg Registration and Organization Online Tool," European Economic Review, Elsevier, vol. 71(C), pages 117-120.
    35. Michal Krawczyk, 2011. "What brings your subjects to the lab? A field experiment," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 482-489, November.
    36. Guillén, Pablo & Veszteg, Róbert F., 2012. "On “lab rats”," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 41(5), pages 714-720.
    37. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    38. Siddharth Suri & Duncan J Watts, 2011. "Cooperation and Contagion in Web-Based, Networked Public Goods Experiments," PLOS ONE, Public Library of Science, vol. 6(3), pages 1-18, March.
    39. repec:cup:judgdm:v:10:y:2015:i:5:p:479-491 is not listed on IDEAS
    40. Akihiro Nishi & Hirokazu Shirado & David G. Rand & Nicholas A. Christakis, 2015. "Inequality and visibility of wealth in experimental social networks," Nature, Nature, vol. 526(7573), pages 426-429, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    2. Thomas Epper & Julien Senn & Ernst Fehr, 2023. "Social preferences across subject pools: students vs. general population," ECON - Working Papers 435, Department of Economics - University of Zurich, revised Jan 2024.
    3. Schmidt, Robert & Schwieren, Christiane & Sproten, Alec N., 2020. "Norms in the lab: Inexperienced versus experienced participants," Journal of Economic Behavior & Organization, Elsevier, vol. 173(C), pages 239-255.
    4. Jon Anderson & Stephen Burks & Jeffrey Carpenter & Lorenz Götte & Karsten Maurer & Daniele Nosenzo & Ruth Potter & Kim Rocha & Aldo Rustichini, 2013. "Self-selection and variations in the laboratory measurement of other-regarding preferences across subject pools: evidence from one college student and two adult samples," Experimental Economics, Springer;Economic Science Association, vol. 16(2), pages 170-189, June.
    5. Stephen V. Burks & Daniele Nosenzo & Jon Anderson & Matthew Bombyk & Derek Ganzhorn & Lorenz Goette & Aldo Rustichini, 2015. "Lab Measures of Other-Regarding Preferences Can Predict Some Related on-the-Job Behavior: Evidence from a Large Scale Field Experiment," Discussion Papers 2015-21, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    6. Ernst Fehr & Thomas Epper & Julien Senn, 2023. "The Missing Type: Where Are the Inequality Averse (Students)?," Working Papers hal-04362826, HAL.
    7. Johannes Abeler & Daniele Nosenzo, 2015. "Self-selection into laboratory experiments: pro-social motives versus monetary incentives," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 195-214, June.
    8. Jonathan Schulz & Uwe Sunde & Petra Thiemann & Christian Thoeni, 2019. "Selection into Experiments: Evidence from a Population of Students," Discussion Papers 2019-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    9. Englmaier, Florian & Gebhardt, Georg, 2016. "Social dilemmas in the laboratory and in the field," Journal of Economic Behavior & Organization, Elsevier, vol. 128(C), pages 85-96.
    10. Marcus Giamattei & Kyanoush Seyed Yahosseini & Simon Gächter & Lucas Molleman, 2020. "LIONESS Lab: a free web-based platform for conducting interactive experiments online," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 95-111, June.
    11. Abeler, Johannes & Nosenzo, Daniele, 2013. "Self-Selection into Economics Experiments Is Driven by Monetary Rewards," IZA Discussion Papers 7374, Institute of Labor Economics (IZA).
    12. Thiemann, Petra & Schulz, Jonathan & Sunde, Uwe & Thöni, Christian, 2022. "Selection into experiments: New evidence on the role of preferences, cognition, and recruitment protocols," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).
    13. Hans-Theo Normann & Till Requate & Israel Waichman, 2014. "Do short-term laboratory experiments provide valid descriptions of long-term economic interactions? A study of Cournot markets," Experimental Economics, Springer;Economic Science Association, vol. 17(3), pages 371-390, September.
    14. Frijters, Paul & Kong, Tao Sherry & Liu, Elaine M., 2015. "Who is coming to the artefactual field experiment? Participation bias among Chinese rural migrants," Journal of Economic Behavior & Organization, Elsevier, vol. 114(C), pages 62-74.
    15. Crawford, Ian & Harris, Donna, 2018. "Social interactions and the influence of “extremists”," Journal of Economic Behavior & Organization, Elsevier, vol. 153(C), pages 238-266.
    16. Weber, Till O. & Schulz, Jonathan F. & Beranek, Benjamin & Lambarraa-Lehnhardt, Fatima & Gächter, Simon, 2023. "The behavioral mechanisms of voluntary cooperation across culturally diverse societies: Evidence from the US, the UK, Morocco, and Turkey," Journal of Economic Behavior & Organization, Elsevier, vol. 215(C), pages 134-152.
    17. Simon Gaechter & Benedikt Herrmann, 2006. "The limits of self-governance in the presence of spite: Experimental evidence from urban and rural Russia," Discussion Papers 2006-13, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    18. Hiroki Ozono & Daisuke Nakama, 2022. "Effects of experimental situation on group cooperation and individual performance: Comparing laboratory and online experiments," PLOS ONE, Public Library of Science, vol. 17(4), pages 1-17, April.
    19. Toke R. Fosgaard, 2018. "Cooperation stability: A representative sample in the lab," IFRO Working Paper 2018/08, University of Copenhagen, Department of Food and Resource Economics.
    20. Schmidt, Robert J. & Schwieren, Christiane & Sproten, Alec N., 2018. "Social Norm Perception in Economic Laboratory Experiments: Inexperienced versus Experienced Participants," Working Papers 0656, University of Heidelberg, Department of Economics.

    More about this item

    Keywords

    experimental methodology; behavioral research; punishment; internet experiments; Amazon Mechanical Turk; public goods game;
    All these keywords.

    JEL classification:

    • C71 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - Cooperative Games
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • D71 - Microeconomics - - Analysis of Collective Decision-Making - - - Social Choice; Clubs; Committees; Associations

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp10517. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.