IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v7y2017i2p2158244017712774.html

Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection

Author

Listed:
  • Logan S. Casey
  • Jesse Chandler
  • Adam Seth Levine
  • Andrew Proctor
  • Dara Z. Strolovitch

Abstract

The online labor market Amazon Mechanical Turk (MTurk) is an increasingly popular source of respondents for social science research. A growing body of research has examined the demographic composition of MTurk workers as compared with that of other populations. While these comparisons have revealed the ways in which MTurk workers are and are not representative of the general population, variations among samples drawn from MTurk have received less attention. This article focuses on whether MTurk sample composition varies as a function of time. Specifically, we examine whether demographic characteristics vary by (a) time of day, (b) day of week, and serial position (i.e., earlier or later in data collection), both (c) across the entire data collection and (d) within specific batches. We find that day of week differences are minimal, but that time of day and serial position are associated with small but important variations in demographic composition. This demonstrates that MTurk samples cannot be presumed identical across different studies, potentially affecting reliability, validity, and efforts to reproduce findings.

Suggested Citation

  • Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    • Handle: RePEc:sae:sagope:v:7:y:2017:i:2:p:2158244017712774
    DOI: 10.1177/2158244017712774
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244017712774
    Download Restriction: no
    File URL: https://libkey.io/10.1177/2158244017712774?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    2. Rajagopal, 2014. "The Human Factors," Palgrave Macmillan Books, in: Architecting Enterprise, chapter 9, pages 225-249, Palgrave Macmillan.
    3. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, "undated". "The Average Laboratory Samples a Population of 7,300 Amazon Mechanical Turk Workers," Mathematica Policy Research Reports f97b669c7b3e4c2ab95c9f805, Mathematica Policy Research.
    4. Jesse Chandler & Danielle Shapiro, "undated". "Conducting Clinical Research Using Crowdsourced Convenience Samples," Mathematica Policy Research Reports c0fd2ad7be9c4bdb8b396aa7e, Mathematica Policy Research.
    5. Robert Peterson & Dwight Merunka, 2014. "Convenience samples of college students and research reproducibility," Post-Print hal-01822317, HAL.
    6. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, 2015. "The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(5), pages 479-491, September.
    7. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    8. Krupnikov, Yanna & Levine, Adam Seth, 2014. "Cross-Sample Comparisons and External Validity," Journal of Experimental Political Science, Cambridge University Press, vol. 1(1), pages 59-80, April.
    9. Peterson, Robert A. & Merunka, Dwight R., 2014. "Convenience samples of college students and research reproducibility," Journal of Business Research, Elsevier, vol. 67(5), pages 1035-1041.
    10. Ravallion, Martin & Lokshin, Michael, 1999. "Subjective economic welfare," Policy Research Working Paper Series 2106, The World Bank.
    11. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    12. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    13. Jesse Chandler & Danielle Shapiro, "undated". "Conducting Clinical Research Using Crowdsourced Convenience Samples," Mathematica Policy Research Reports c9ae2ea1c9b249deadb0c7c0d, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dulisse, Brandon Christopher & Connealy, Nathan & Logan, Matthew William, 2024. "“Get rich quick,” scheme or script? The effect of cryptoculture on the susceptibility of fraud victimization among cryptocurrency purchasers," Journal of Criminal Justice, Elsevier, vol. 94(C).
    2. Kusterer, David & Sliwka, Dirk, 2022. "Social Preferences and Rating Biases in Subjective Performance Evaluations," IZA Discussion Papers 15496, IZA Network @ LISER.
    3. Chapkovski, Philipp, 2023. "Conducting interactive experiments on Toloka," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    4. Florian Teschner & Henner Gimpel, 2018. "Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments," Group Decision and Negotiation, Springer, vol. 27(2), pages 197-214, April.
    5. David J. Kusterer & Dirk Sliwka, 2025. "Social Preferences and the Informativeness of Subjective Performance Evaluations," Management Science, INFORMS, vol. 71(5), pages 4028-4048, May.
    6. Andrew J. Thompson & Justin T. Pickett, 2020. "Are Relational Inferences from Crowdsourced and Opt-in Samples Generalizable? Comparing Criminal Justice Attitudes in the GSS and Five Online Samples," Journal of Quantitative Criminology, Springer, vol. 36(4), pages 907-932, December.
    7. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    8. Kazakova, E. & Sandomirskaia, M. & Suvorov, A. & Khazhgerieva, A. & Shavshin, R., 2023. "Platforms, online labor markets, and crowdsourcing. Part 2. Crowdsourcing," Journal of the New Economic Association, New Economic Association, vol. 61(4), pages 128-144.
    9. Crispin Cooper & Ana Fredrich & Tommaso Reggiani & Wouter Poortinga, 2025. "Individual utilities of life satisfaction reveal inequality aversion unrelated to political alignment," Papers 2509.07793, arXiv.org, revised Oct 2025.
    10. R. Gordon Rinderknecht, 2019. "Effects of Participant Displeasure on the Social-Psychological Study of Power on Amazon’s Mechanical Turk," SAGE Open, , vol. 9(3), pages 21582440198, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    2. Schwaiger, Rene & Hueber, Laura, 2021. "Do MTurkers exhibit myopic loss aversion?," Economics Letters, Elsevier, vol. 209(C).
    3. Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    4. Blaine G. Robbins, 2017. "Status, identity, and ability in the formation of trust," Rationality and Society, , vol. 29(4), pages 408-448, November.
    5. Austin M Strange & Ryan D Enos & Mark Hill & Amy Lakeman, 2019. "Online volunteer laboratories for human subjects research," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-13, August.
    6. David Ronayne & Daniel Sgroi, 2018. "On the motivations for the dual-use of electronic and traditional cigarettes," Applied Economics Letters, Taylor & Francis Journals, vol. 25(12), pages 830-834, July.
    7. Christ, Margaret H. & Vance, Thomas W., 2018. "Cascading controls: The effects of managers’ incentives on subordinate effort to help or harm," Accounting, Organizations and Society, Elsevier, vol. 65(C), pages 20-32.
    8. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    9. Roman Lukyanenko & Andrea Wiggins & Holly K. Rosser, 0. "Citizen Science: An Information Quality Research Frontier," Information Systems Frontiers, Springer, vol. 0, pages 1-23.
    10. Capraro, Valerio & Schulz, Jonathan & Rand, David G., 2019. "Time pressure and honesty in a deception game," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 79(C), pages 93-99.
    11. Abel François & Sophie Panel & Laurent Weill, 2023. "Dictators’ facial characteristics and foreign direct investment," Post-Print hal-03969697, HAL.
    12. David Johnson & John Barry Ryan, 2020. "Amazon Mechanical Turk workers can provide consistent and economically meaningful data," Southern Economic Journal, John Wiley & Sons, vol. 87(1), pages 369-385, July.
    13. repec:plo:pone00:0198835 is not listed on IDEAS
    14. Xiaochuan Song & Graham H. Lowman & Peter Harms, 2020. "Justice for the Crowd: Organizational Justice and Turnover in Crowd-Based Labor," Administrative Sciences, MDPI, vol. 10(4), pages 1-37, November.
    15. Bhatt, Vipul & Smith, Angela M., 2025. "Overconfidence and performance: Evidence from a simple real-effort task," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 114(C).
    16. Shari De Baets & Dilek Önkal & Wasim Ahmed, 2022. "Do Risky Scenarios Affect Forecasts of Savings and Expenses?," Forecasting, MDPI, vol. 4(1), pages 1-28, February.
    17. Palan, Stefan & Schitter, Christian, 2018. "Prolific.ac—A subject pool for online experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 17(C), pages 22-27.
    18. Rene Schwaiger & Laura Hueber, 2021. "Do MTurkers Exhibit Myopic Loss Aversion?," Working Papers 2021-12, Faculty of Economics and Statistics, Universität Innsbruck.
    19. Keela S. Thomson & Daniel M. Oppenheimer, 2016. "Investigating an alternate form of the cognitive reflection test," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 11(1), pages 99-113, January.
    20. repec:plo:pone00:0226394 is not listed on IDEAS
    21. Yoshimasa Majima, 2017. "The Feasibility of a Japanese Crowdsourcing Service for Experimental Research in Psychology," SAGE Open, , vol. 7(1), pages 21582440176, March.
    22. Christoph Bartneck & Andreas Duenser & Elena Moltchanova & Karolina Zawieska, 2015. "Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-23, April.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:7:y:2017:i:2:p:2158244017712774. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.