IDEAS home Printed from https://ideas.repec.org/a/jdm/journl/v10y2015i5p479-491.html
   My bibliography  Save this article

The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

Author

Listed:
  • Neil Stewart
  • Christoph Ungemach
  • Adam J. L. Harris
  • Daniel M. Bartels
  • Ben R. Newell
  • Gabriele Paolacci
  • Jesse Chandler

Abstract

Using capture-recapture analysis we estimate the effective size of the active Amazon Mechanical Turk (MTurk) population that a typical laboratory can access to be about 7,300 workers. We also estimate that the time taken for half of the workers to leave the MTurk pool and be replaced is about 7 months. Each laboratory has its own population pool which overlaps, often extensively, with the hundreds of other laboratories using MTurk. Our estimate is based on a sample of 114,460 completed sessions from 33,408 unique participants and 689 sessions across seven laboratories in the US, Europe, and Australia from January 2012 to March 2015.

Suggested Citation

  • Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, 2015. "The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(5), pages 479-491, September.
    • Handle: RePEc:jdm:journl:v:10:y:2015:i:5:p:479-491
    as

    Download full text from publisher

    File URL: http://journal.sjdm.org/14/14725/jdm14725.pdf
    Download Restriction: no
    File URL: http://journal.sjdm.org/14/14725/jdm14725.html
    Download Restriction: no

    References listed on IDEAS

    as
    1. Baillargeon, Sophie & Rivest, Louis-Paul, 2007. "Rcapture: Loglinear Models for Capture-Recapture in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 19(i05).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Capraro, Valerio & Schulz, Jonathan & Rand, David G., 2019. "Time pressure and honesty in a deception game," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 79(C), pages 93-99.
    2. Stevenson, Regan M. & Josefy, Matthew, 2019. "Knocking at the gate: The path to publication for entrepreneurship experiments through the lens of gatekeeping theory," Journal of Business Venturing, Elsevier, vol. 34(2), pages 242-260.
    3. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    4. Jonathan Robinson & Cheskie Rosenzweig & Aaron J Moss & Leib Litman, 2019. "Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-29, December.
    5. Christ, Margaret H. & Vance, Thomas W., 2018. "Cascading controls: The effects of managers’ incentives on subordinate effort to help or harm," Accounting, Organizations and Society, Elsevier, vol. 65(C), pages 20-32.
    6. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    7. Brandi S. Morris & Polymeros Chrysochou & Jacob Dalgaard Christensen & Jacob L. Orquin & Jorge Barraza & Paul J. Zak & Panagiotis Mitkidis, 2019. "Stories vs. facts: triggering emotion and action-taking on climate change," Climatic Change, Springer, vol. 154(1), pages 19-36, May.
    8. Clément Le Ludec & Paola Tubaro & Antonio Casilli, 2019. "How many people microwork in France? Estimating the size of a new labor force," Working Papers hal-02012731, HAL.
    9. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2019. "Geographical Concentration and Editorial Favoritism within the Field of Laboratory Experimental Economics," Research Memorandum 029, Maastricht University, Graduate School of Business and Economics (GSBE).
    10. Palan, Stefan & Schitter, Christian, 2018. "Prolific.ac—A subject pool for online experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 17(C), pages 22-27.
    11. Surti, Chirag & Celani, Anthony & Gajpal, Yuvraj, 2020. "The newsvendor problem: The role of prospect theory and feedback," European Journal of Operational Research, Elsevier, vol. 287(1), pages 251-261.
    12. Anthony M. Evans & Joachim I. Krueger, 2017. "Ambiguity and expectation-neglect in dilemmas of interpersonal trust," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(6), pages 584-595, November.
    13. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2020. "Geographical Concentration and Editorial Favoritism within the Field of Laboratory Experimental Economics (RM/19/029-revised-)," Research Memorandum 014, Maastricht University, Graduate School of Business and Economics (GSBE).
    14. David Ronayne & Daniel Sgroi, 2018. "On the motivations for the dual-use of electronic and traditional cigarettes," Applied Economics Letters, Taylor & Francis Journals, vol. 25(12), pages 830-834, July.
    15. Clément Le Ludec & Paola Tubaro & Antonio Casilli, 2019. "Combien de personnes micro-travaillent en France ? Estimer l'ampleur d'une nouvelle forme de travail," Working Papers hal-02021525, HAL.
    16. Keela S. Thomson & Daniel M. Oppenheimer, 2016. "Investigating an alternate form of the cognitive reflection test," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 11(1), pages 99-113, January.
    17. Summer Allen & Aaron M. McCright & Thomas Dietz, 2017. "A Social Movement Identity Instrument for Integrating Survey Methods Into Social Movements Research," SAGE Open, , vol. 7(2), pages 21582440177, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Donald T McKnight & Day B Ligon, 2017. "Correcting for unequal catchability in sex ratio and population size estimates," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-13, August.
    2. Jolynn Pek & Hao Wu, 2015. "Profile Likelihood-Based Confidence Intervals and Regions for Structural Equation Models," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 1123-1145, December.
    3. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, "undated". "The Average Laboratory Samples a Population of 7,300 Amazon Mechanical Turk Workers," Mathematica Policy Research Reports f97b669c7b3e4c2ab95c9f805, Mathematica Policy Research.
    4. Daniel Manrique‐Vallier, 2016. "Bayesian population size estimation using Dirichlet process mixtures," Biometrics, The International Biometric Society, vol. 72(4), pages 1246-1254, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jdm:journl:v:10:y:2015:i:5:p:479-491. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jonathan Baron). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.