IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/111980.html
   My bibliography  Save this paper

Interactive experiments in Toloka

Author

Listed:
  • Chapkovski, Philipp

Abstract

The popularity of online behavioral experiments grew steadily even before the COVID-19 pandemic. With the start of lockdowns, online studies were often the only available option for the behavioral economists, sociologists and political scientists. The usage of most well-known platforms such as mTurk was so intensive that it harmed the quality of data. But even before the pandemics-induced quality crisis, online studies were limited in scope, since real-time interactions between participants were hard to achieve due to the large proportion of drop-outs and issues with creating stable groups. Using the crowdsourcing platform Toloka, we successfully ran several multi-round interactive experiments. Toloka’s large online audience, relatively low exposure of participants to sociological surveys and behavioral studies, and a convenient application programming interface makes it a perfect tool to run behavioral studies that require real-time interactions of participants.

Suggested Citation

  • Chapkovski, Philipp, 2022. "Interactive experiments in Toloka," MPRA Paper 111980, University Library of Munich, Germany.
    • Handle: RePEc:pra:mprapa:111980
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/111980/1/MPRA_paper_111980.pdf
    File Function: original version
    Download Restriction: no
    File URL: https://mpra.ub.uni-muenchen.de/116440/1/MPRA_paper_116440.pdf
    File Function: revised version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Timothy N. Cason & Daniel Friedman & ED Hopkins, 2014. "Cycles and Instability in a Rock--Paper--Scissors Population Game: A Continuous Time Experiment," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(1), pages 112-136.
    2. Charness, Gary & Gneezy, Uri & Halladay, Brianna, 2016. "Experimental methods: Pay one or pay all," Journal of Economic Behavior & Organization, Elsevier, vol. 131(PA), pages 141-150.
    3. Heinz, Matthias & Juranek, Steffen & Rau, Holger A., 2012. "Do women behave more reciprocally than men? Gender differences in real effort dictator games," Journal of Economic Behavior & Organization, Elsevier, vol. 83(1), pages 105-110.
    4. Ofra Amir & David G Rand & Ya'akov Kobi Gal, 2012. "Economic Games on the Internet: The Effect of $1 Stakes," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-4, February.
    5. Erin L. Krupka & Roberto A. Weber, 2013. "Identifying Social Norms Using Coordination Games: Why Does Dictator Game Sharing Vary?," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 495-524, June.
    6. David J. Cooper & Hanming Fang, 2008. "Understanding Overbidding in Second Price Auctions: An Experimental Study," Economic Journal, Royal Economic Society, vol. 118(532), pages 1572-1595, October.
    7. Gächter, Simon & Herrmann, Benedikt, 2011. "The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia," European Economic Review, Elsevier, vol. 55(2), pages 193-210, February.
    8. Andreas Bergh & Philipp C Wichardt, 2022. "Mine or ours? Unintended framing effects in dictator games," Rationality and Society, , vol. 34(1), pages 78-95, February.
    9. Volker Benndorf & Claudia Moellers & Hans-Theo Normann, 2017. "Experienced vs. inexperienced participants in the lab: do they behave differently?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 12-25, July.
    10. Erik Snowberg & Leeat Yariv, 2021. "Testing the Waters: Behavior across Participant Pools," American Economic Review, American Economic Association, vol. 111(2), pages 687-719, February.
    11. Muriel Niederle & Lise Vesterlund, 2007. "Do Women Shy Away From Competition? Do Men Compete Too Much?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1067-1101.
    12. Nagel, Rosemarie, 1995. "Unraveling in Guessing Games: An Experimental Study," American Economic Review, American Economic Association, vol. 85(5), pages 1313-1326, December.
    13. Benndorf, Volker & Moellers, Claudia & Normann, Hans-Theo, 2017. "Experienced vs. inexperienced participants in the lab: Do they behave differently?," DICE Discussion Papers 251, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    14. Joseph Henrich & Steve J. Heine & Ara Norenzayan, 2010. "The Weirdest People in the World?," RatSWD Working Papers 139, German Data Forum (RatSWD).
    15. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.
    16. Andreoni, James, 1988. "Why free ride? : Strategies and learning in public goods experiments," Journal of Public Economics, Elsevier, vol. 37(3), pages 291-304, December.
    17. Hugh-Jones, David, 2016. "Honesty, beliefs about honesty, and economic growth in 15 countries," Journal of Economic Behavior & Organization, Elsevier, vol. 127(C), pages 99-114.
    18. repec:cup:judgdm:v:10:y:2015:i:5:p:479-491 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chapkovski, Philipp & Zakharov, Alexei, 2023. "Does voluntary disclosure of polarizing information make polarization deeper? An online experiment on Russo-Ukrainian War," MPRA Paper 116305, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chapkovski, Philipp, 2022. "Interactive Experiments in Toloka," EconStor Preprints 249771, ZBW - Leibniz Information Centre for Economics.
    2. Chapkovski, Philipp, 2023. "Conducting interactive experiments on Toloka," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    3. Benndorf, Volker & Rau, Holger A. & Sölch, Christian, 2019. "Minimizing learning in repeated real-effort tasks," Journal of Behavioral and Experimental Finance, Elsevier, vol. 22(C), pages 239-248.
    4. Roggenkamp, Hauke C., 2024. "Revisiting ‘Growth and Inequality in Public Good Provision’—Reproducing and Generalizing Through Inconvenient Online Experimentation," OSF Preprints 6rn97, Center for Open Science.
    5. Thiemann, Petra & Schulz, Jonathan & Sunde, Uwe & Thöni, Christian, 2022. "Selection into experiments: New evidence on the role of preferences, cognition, and recruitment protocols," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).
    6. Arechar, Antonio A. & Rand, David G., 2022. "Learning to be selfish? A large-scale longitudinal analysis of Dictator games played on Amazon Mechanical Turk," Journal of Economic Psychology, Elsevier, vol. 90(C).
    7. Dai, Zhixin & Zheng, Jiwei & Zizzo, Daniel John, 2024. "Theories of reasoning and focal point play with a matched non-student sample," China Economic Review, Elsevier, vol. 83(C).
    8. Tiziana Medda & Vittorio Pelligra & Tommaso Reggiani, 2021. "Lab-Sophistication: Does Repeated Participation in Laboratory Experiments Affect Pro-Social Behaviour?," Games, MDPI, vol. 12(1), pages 1-14, February.
    9. Gruner, Sven & Lehberger, Mira & Hirschauer, Norbert & Mußhoff, Oliver, 2022. "How (un)informative are experiments with students for other social groups? A study of agricultural students and farmers," Australian Journal of Agricultural and Resource Economics, Australian Agricultural and Resource Economics Society, vol. 66(03), January.
    10. Dengler-Roscher, Kathrin & Montinari, Natalia & Panganiban, Marian & Ploner, Matteo & Werner, Benedikt, 2018. "On the malleability of fairness ideals: Spillover effects in partial and impartial allocation tasks," Journal of Economic Psychology, Elsevier, vol. 65(C), pages 60-74.
    11. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    12. Fortuna Casoria & Fabio Galeotti & Marie Claire Villeval, 2024. "Trust and Social Preferences in Times of Acute Health Crisis," Annals of Economics and Statistics, GENES, issue 154, pages 5-50.
    13. Schmidt, Robert & Schwieren, Christiane & Sproten, Alec N., 2020. "Norms in the lab: Inexperienced versus experienced participants," Journal of Economic Behavior & Organization, Elsevier, vol. 173(C), pages 239-255.
    14. Gary Bolton & Eugen Dimant & Ulrich Schmidt, 2018. "When a Nudge Backfires. Using Observation with Social and Economic Incentives to Promote Pro-Social Behavior," PPE Working Papers 0017, Philosophy, Politics and Economics, University of Pennsylvania.
    15. Gächter, Simon & Lee, Kyeongtae & Sefton, Martin & Weber, Till O., 2024. "The role of payoff parameters for cooperation in the one-shot Prisoner's Dilemma," European Economic Review, Elsevier, vol. 166(C).
    16. repec:hal:wpaper:hal-03974756 is not listed on IDEAS
    17. Palan, Stefan & Schitter, Christian, 2018. "Prolific.ac—A subject pool for online experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 17(C), pages 22-27.
    18. Schram, Arthur & Zheng, Jin Di & Zhuravleva, Tatyana, 2022. "Corruption: A cross-country comparison of contagion and conformism," Journal of Economic Behavior & Organization, Elsevier, vol. 193(C), pages 497-518.
    19. David Hugh-Jones, 2019. "True lies," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 5(2), pages 255-268, December.
    20. Akdeniz, Aslıhan & van Veelen, Matthijs, 2023. "Evolution and the ultimatum game," Games and Economic Behavior, Elsevier, vol. 142(C), pages 570-612.
    21. Aurélie Dariel & Nikos Nikiforakis & Jan Stoop, 2020. "Does selection bias cause us to overestimate gender differences in competitiveness?," Working Papers 20200046, New York University Abu Dhabi, Department of Social Science, revised May 2020.

    More about this item

    Keywords

    Crowdsourcing; mTurk; online research; survey research;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:111980. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.