IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0226394.html
   My bibliography  Save this article

Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool

Author

Listed:
  • Jonathan Robinson
  • Cheskie Rosenzweig
  • Aaron J Moss
  • Leib Litman

Abstract

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk’s utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool. Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

Suggested Citation

  • Jonathan Robinson & Cheskie Rosenzweig & Aaron J Moss & Leib Litman, 2019. "Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-29, December.
  • Handle: RePEc:plo:pone00:0226394
    DOI: 10.1371/journal.pone.0226394
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0226394
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0226394&type=printable
    Download Restriction: no

    References listed on IDEAS

    as
    1. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, 2015. "The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(5), pages 479-491, September.
    2. Jesse Chandler & Gabriele Paolacci & Eyal Peer & Pam Mueller & Kate A. Ratliff, 2015. "Using Nonnaive Participants Can Reduce Effect Sizes," Mathematica Policy Research Reports bffac982a56e4cfba3659e74a, Mathematica Policy Research.
    3. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, "undated". "The Average Laboratory Samples a Population of 7,300 Amazon Mechanical Turk Workers," Mathematica Policy Research Reports f97b669c7b3e4c2ab95c9f805, Mathematica Policy Research.
    4. Matthew J C Crump & John V McDonnell & Todd M Gureckis, 2013. "Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research," PLOS ONE, Public Library of Science, vol. 8(3), pages 1-18, March.
    5. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sanjit Dhami & Narges Hajimoladarvish, 2020. "Mental Accounting, Loss Aversion, and Tax Evasion: Theory and Evidence," CESifo Working Paper Series 8606, CESifo.
    2. Benjamin Enke & Thomas Graeber, 2019. "Cognitive Uncertainty," CESifo Working Paper Series 7971, CESifo.
    3. Paola Tubaro & Clément Le Ludec & Antonio Casilli, 2020. "Counting ‘micro-workers’: societal and methodological challenges around new forms of labour," Post-Print hal-02898905, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Keela S. Thomson & Daniel M. Oppenheimer, 2016. "Investigating an alternate form of the cognitive reflection test," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 11(1), pages 99-113, January.
    2. Capraro, Valerio & Schulz, Jonathan & Rand, David G., 2019. "Time pressure and honesty in a deception game," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 79(C), pages 93-99.
    3. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    4. Anthony M. Evans & Joachim I. Krueger, 2017. "Ambiguity and expectation-neglect in dilemmas of interpersonal trust," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(6), pages 584-595, November.
    5. Cl'ement Le Ludec & Paola Tubaro & Antonio A. Casilli, 2019. "How many people microwork in France? Estimating the size of a new labor force," Papers 1901.03889, arXiv.org.
    6. David Ronayne & Daniel Sgroi, 2018. "On the motivations for the dual-use of electronic and traditional cigarettes," Applied Economics Letters, Taylor & Francis Journals, vol. 25(12), pages 830-834, July.
    7. Brandi S. Morris & Polymeros Chrysochou & Jacob Dalgaard Christensen & Jacob L. Orquin & Jorge Barraza & Paul J. Zak & Panagiotis Mitkidis, 2019. "Stories vs. facts: triggering emotion and action-taking on climate change," Climatic Change, Springer, vol. 154(1), pages 19-36, May.
    8. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2020. "Geographical Concentration and Editorial Favoritism within the Field of Laboratory Experimental Economics (RM/19/029-revised-)," Research Memorandum 014, Maastricht University, Graduate School of Business and Economics (GSBE).
    9. Christ, Margaret H. & Vance, Thomas W., 2018. "Cascading controls: The effects of managers’ incentives on subordinate effort to help or harm," Accounting, Organizations and Society, Elsevier, vol. 65(C), pages 20-32.
    10. Palan, Stefan & Schitter, Christian, 2018. "Prolific.ac—A subject pool for online experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 17(C), pages 22-27.
    11. Clément Le Ludec & Paola Tubaro & Antonio Casilli, 2019. "Combien de personnes micro-travaillent en France ? Estimer l'ampleur d'une nouvelle forme de travail," Working Papers hal-02021525, HAL.
    12. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.
    13. Stevenson, Regan M. & Josefy, Matthew, 2019. "Knocking at the gate: The path to publication for entrepreneurship experiments through the lens of gatekeeping theory," Journal of Business Venturing, Elsevier, vol. 34(2), pages 242-260.
    14. Cooper, David J. & Saral, Krista & Villeval, Marie Claire, 2019. "Why Join a Team?," IZA Discussion Papers 12587, Institute of Labor Economics (IZA).
      • David J. Cooper & Krista Saral & Marie Claire Villeval, 2021. "Why join teams?," Post-Print halshs-03003653, HAL.
      • David Cooper & Krista Saral & Marie Claire Villeval, 2019. "Why Join a Team?," Working Papers halshs-02295921, HAL.
      • David J. Cooper & Krista Saral & Marie Claire Villeval, 2019. "Why Join a Team?," Working Papers 1928, Groupe d'Analyse et de Théorie Economique Lyon St-Étienne (GATE Lyon St-Étienne), Université de Lyon.
    15. Zakaria Babutsidze & Nobuyuki Hanaki & Adam Zylbersztejn, 2019. "Digital Communication and Swift Trust," Post-Print halshs-02409314, HAL.
    16. Chavez, Daniel E. & Palma, Marco A. & Nayga, Rodolfo M. & Mjelde, James W., 2020. "Product availability in discrete choice experiments with private goods," Journal of choice modelling, Elsevier, vol. 36(C).
    17. Prokudina, Elena & Renneboog, Luc & Tobler, Philippe, 2015. "Does Confidence Predict Out-of-Domain Effort?," Discussion Paper 2015-055, Tilburg University, Center for Economic Research.
    18. Noussair, C.N. & Tucker, S. & Xu, Yilong, 2014. "A Future Market Reduces Bubbles but Allows Greater Profit for More Sophisticated Traders," Other publications TiSEM 43ded173-9eee-48a4-8a15-6, Tilburg University, School of Economics and Management.
    19. Francis Bloch & Bhaskar Dutta & Stéphane Robin & Min Zhu, 2016. "The formation of partnerships in social networks," Post-Print halshs-01421347, HAL.
    20. Corgnet, Brice & DeSantis, Mark & Porter, David, 2020. "The distribution of information and the price efficiency of markets," Journal of Economic Dynamics and Control, Elsevier, vol. 110(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0226394. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (plosone). General contact details of provider: https://journals.plos.org/plosone/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.