IDEAS home Printed from https://ideas.repec.org/a/eee/beexfi/v37y2023ics2214635023000047.html
   My bibliography  Save this article

Conducting interactive experiments on Toloka

Author

Listed:
  • Chapkovski, Philipp

Abstract

Even before the COVID-19 pandemic, the popularity of online behavioral experiments grew steadily. Due to lockdowns, online studies often became the only available option for behavioral economists, sociologists, and political scientists. The use of the most well-known platforms, such as mTurk, was so intensive that the quality of data was harmed. However, even before the pandemic-induced quality crisis, online studies were limited in scope; real-time interactions between participants were hard to achieve due to the large proportion of drop-outs and issues with creating stable groups. Using the relatively unknown crowdsourcing platform, Toloka, we successfully ran several multi-round interactive experiments. Toloka’s sizeable online population, fairly low exposure of participants to sociological surveys and behavioral studies, and convenient application programming interface can make it a useful addition to the toolbox of an experimentalist who needs to run behavioral studies that require real-time interactions between participants.

Suggested Citation

  • Chapkovski, Philipp, 2023. "Conducting interactive experiments on Toloka," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    • Handle: RePEc:eee:beexfi:v:37:y:2023:i:c:s2214635023000047
    DOI: 10.1016/j.jbef.2023.100790
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S2214635023000047
    Download Restriction: no
    File URL: https://libkey.io/10.1016/j.jbef.2023.100790?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jordi Brandts & Gary Charness, 2011. "The strategy versus the direct-response method: a first survey of experimental comparisons," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 375-398, September.
    2. DavidJ. Cooper & Hanming Fang, 2008. "Understanding Overbidding In Second Price Auctions: An Experimental Study," Economic Journal, Royal Economic Society, vol. 118(532), pages 1572-1595, October.
    3. Fischbacher, Urs & Gächter, Simon & Quercia, Simone, 2012. "The behavioral validity of the strategy method in public good experiments," Journal of Economic Psychology, Elsevier, vol. 33(4), pages 897-913.
    4. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    5. Johannes Abeler & Armin Falk & Lorenz Goette & David Huffman, 2011. "Reference Points and Effort Provision," American Economic Review, American Economic Association, vol. 101(2), pages 470-492, April.
    6. Jennifer Zelmer, 2003. "Linear Public Goods Experiments: A Meta-Analysis," Experimental Economics, Springer;Economic Science Association, vol. 6(3), pages 299-310, November.
    7. R. Mark Isaac & James M. Walker, 1988. "Group Size Effects in Public Goods Provision: The Voluntary Contributions Mechanism," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 103(1), pages 179-199.
    8. Erin L. Krupka & Roberto A. Weber, 2013. "Identifying Social Norms Using Coordination Games: Why Does Dictator Game Sharing Vary?," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 495-524, June.
    9. Gächter, Simon & Herrmann, Benedikt, 2011. "The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia," European Economic Review, Elsevier, vol. 55(2), pages 193-210, February.
    10. Muriel Niederle & Lise Vesterlund, 2007. "Do Women Shy Away From Competition? Do Men Compete Too Much?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1067-1101.
    11. Christoph Engel, 2011. "Dictator games: a meta study," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 583-610, November.
    12. Benndorf, Volker & Moellers, Claudia & Normann, Hans-Theo, 2017. "Experienced vs. inexperienced participants in the lab: Do they behave differently?," DICE Discussion Papers 251, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    13. Simon Gachter & Ernst Fehr, 2000. "Cooperation and Punishment in Public Goods Experiments," American Economic Review, American Economic Association, vol. 90(4), pages 980-994, September.
    14. Joseph Henrich & Steve J. Heine & Ara Norenzayan, 2010. "The Weirdest People in the World?," RatSWD Working Papers 139, German Data Forum (RatSWD).
    15. Abeler, Johannes & Becker, Anke & Falk, Armin, 2014. "Representative evidence on lying costs," Journal of Public Economics, Elsevier, vol. 113(C), pages 96-104.
    16. Yun Shin Lee & Yong Won Seo & Enno Siemsen, 2018. "Running Behavioral Operations Experiments Using Amazon's Mechanical Turk," Production and Operations Management, Production and Operations Management Society, vol. 27(5), pages 973-989, May.
    17. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.
    18. Volker Benndorf & Claudia Moellers & Hans-Theo Normann, 2017. "Experienced vs. inexperienced participants in the lab: do they behave differently?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 12-25, July.
    19. Hugh-Jones, David, 2016. "Honesty, beliefs about honesty, and economic growth in 15 countries," Journal of Economic Behavior & Organization, Elsevier, vol. 127(C), pages 99-114.
    20. Erik Snowberg & Leeat Yariv, 2021. "Testing the Waters: Behavior across Participant Pools," American Economic Review, American Economic Association, vol. 111(2), pages 687-719, February.
    21. repec:cup:judgdm:v:10:y:2015:i:5:p:479-491 is not listed on IDEAS
    22. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hamza Umer, 2024. "Covid-19 and altruism: a meta-analysis of dictator games," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 51(1), pages 35-60, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chapkovski, Philipp, 2022. "Interactive Experiments in Toloka," EconStor Preprints 249771, ZBW - Leibniz Information Centre for Economics.
    2. Chapkovski, Philipp, 2022. "Interactive experiments in Toloka," MPRA Paper 111980, University Library of Munich, Germany.
    3. Roggenkamp, Hauke C., 2024. "Revisiting ‘Growth and Inequality in Public Good Provision’—Reproducing and Generalizing Through Inconvenient Online Experimentation," OSF Preprints 6rn97, Center for Open Science.
    4. Eugen Dimant & Tobias Gesche, 2021. "Nudging Enforcers: How Norm Perceptions and Motives for Lying Shape Sanctions," CESifo Working Paper Series 9385, CESifo.
    5. Thiemann, Petra & Schulz, Jonathan & Sunde, Uwe & Thöni, Christian, 2022. "Selection into experiments: New evidence on the role of preferences, cognition, and recruitment protocols," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).
    6. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    7. Daniel Jones & Sera Linardi, 2014. "Wallflowers: Experimental Evidence of an Aversion to Standing Out," Management Science, INFORMS, vol. 60(7), pages 1757-1771, July.
    8. Arechar, Antonio A. & Rand, David G., 2022. "Learning to be selfish? A large-scale longitudinal analysis of Dictator games played on Amazon Mechanical Turk," Journal of Economic Psychology, Elsevier, vol. 90(C).
    9. Riehm, Tobias & Fugger, Nicolas & Gillen, Philippe & Gretschko, Vitali & Werner, Peter, 2022. "Social norms, sanctions, and conditional entry in markets with externalities: Evidence from an artefactual field experiment," Journal of Public Economics, Elsevier, vol. 212(C).
    10. Bartoš, Vojtěch, 2021. "Seasonal scarcity and sharing norms," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 303-316.
    11. repec:cup:judgdm:v:16:y:2021:i:5:p:1267-1289 is not listed on IDEAS
    12. Bryan C. McCannon & Paul Walker, 2020. "Individual Competence and Committee Decision Making: Experimental Evidence," Southern Economic Journal, John Wiley & Sons, vol. 86(4), pages 1531-1558, April.
    13. Karakostas, Alexandros & Kocher, Martin G. & Matzat, Dominik & Rau, Holger A. & Riewe, Gerhard, 2023. "The team allocator game: Allocation power in public goods games," Games and Economic Behavior, Elsevier, vol. 140(C), pages 73-87.
    14. Erik O. Kimbrough & Alexander Vostroknutov, 2016. "Norms Make Preferences Social," Journal of the European Economic Association, European Economic Association, vol. 14(3), pages 608-638, June.
    15. Karakostas, Alexandros & Kocher, Martin & Matzat, Dominik & Rau, Holger A. & Riewe, Gerhard, 2021. "The team allocator game: Allocation power in public goods games," University of Göttingen Working Papers in Economics 419, University of Goettingen, Department of Economics.
    16. Breitmoser, Yves & Vorjohann, Pauline, 2018. "Welfare-Based Altruism," Rationality and Competition Discussion Paper Series 89, CRC TRR 190 Rationality and Competition.
    17. Lilia Zhurakhovska, 2014. "Strategic Trustworthiness via Unstrategic Third-party Reward – An Experiment," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2014_06, Max Planck Institute for Research on Collective Goods, revised Jan 2017.
    18. Columbus, Simon & Böhm, Robert, 2021. "Norm shifts under the strategy method," Judgment and Decision Making, Cambridge University Press, vol. 16(5), pages 1267-1289, September.
    19. Simon Gaechter, 2014. "Human Pro-Social Motivation and the Maintenance of Social Order," Discussion Papers 2014-02, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    20. Sebastian J. Goerg & David Rand & Gari Walkowitz, 2020. "Framing effects in the prisoner’s dilemma but not in the dictator game," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 1-12, June.
    21. Felix Kölle & Thomas Lauer, 2020. "Understanding Cooperation in an Intertemporal Context," ECONtribute Discussion Papers Series 046, University of Bonn and University of Cologne, Germany.

    More about this item

    Keywords

    Crowdsourcing; mTurk; Online research; Survey research;
    All these keywords.

    JEL classification:

    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:beexfi:v:37:y:2023:i:c:s2214635023000047. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/journal-of-behavioral-and-experimental-finance .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.