IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v7y2017i2p2158244017712774.html
   My bibliography  Save this article

Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection

Author

Listed:
  • Logan S. Casey
  • Jesse Chandler
  • Adam Seth Levine
  • Andrew Proctor
  • Dara Z. Strolovitch

Abstract

The online labor market Amazon Mechanical Turk (MTurk) is an increasingly popular source of respondents for social science research. A growing body of research has examined the demographic composition of MTurk workers as compared with that of other populations. While these comparisons have revealed the ways in which MTurk workers are and are not representative of the general population, variations among samples drawn from MTurk have received less attention. This article focuses on whether MTurk sample composition varies as a function of time. Specifically, we examine whether demographic characteristics vary by (a) time of day, (b) day of week, and serial position (i.e., earlier or later in data collection), both (c) across the entire data collection and (d) within specific batches. We find that day of week differences are minimal, but that time of day and serial position are associated with small but important variations in demographic composition. This demonstrates that MTurk samples cannot be presumed identical across different studies, potentially affecting reliability, validity, and efforts to reproduce findings.

Suggested Citation

  • Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    • Handle: RePEc:sae:sagope:v:7:y:2017:i:2:p:2158244017712774
    DOI: 10.1177/2158244017712774
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244017712774
    Download Restriction: no
    File URL: https://libkey.io/10.1177/2158244017712774?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    2. Rajagopal, 2014. "The Human Factors," Palgrave Macmillan Books, in: Architecting Enterprise, chapter 9, pages 225-249, Palgrave Macmillan.
    3. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, "undated". "The Average Laboratory Samples a Population of 7,300 Amazon Mechanical Turk Workers," Mathematica Policy Research Reports f97b669c7b3e4c2ab95c9f805, Mathematica Policy Research.
    4. Jesse Chandler & Danielle Shapiro, "undated". "Conducting Clinical Research Using Crowdsourced Convenience Samples," Mathematica Policy Research Reports c0fd2ad7be9c4bdb8b396aa7e, Mathematica Policy Research.
    5. Robert Peterson & Dwight Merunka, 2014. "Convenience samples of college students and research reproducibility," Post-Print hal-01822317, HAL.
    6. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, 2015. "The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(5), pages 479-491, September.
    7. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    8. Krupnikov, Yanna & Levine, Adam Seth, 2014. "Cross-Sample Comparisons and External Validity," Journal of Experimental Political Science, Cambridge University Press, vol. 1(1), pages 59-80, April.
    9. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    10. Peterson, Robert A. & Merunka, Dwight R., 2014. "Convenience samples of college students and research reproducibility," Journal of Business Research, Elsevier, vol. 67(5), pages 1035-1041.
    11. Ravallion, Martin & Lokshin, Michael, 1999. "Subjective economic welfare," Policy Research Working Paper Series 2106, The World Bank.
    12. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    13. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    14. repec:cup:judgdm:v:10:y:2015:i:5:p:479-491 is not listed on IDEAS
    15. Jesse Chandler & Danielle Shapiro, "undated". "Conducting Clinical Research Using Crowdsourced Convenience Samples," Mathematica Policy Research Reports c9ae2ea1c9b249deadb0c7c0d, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dulisse, Brandon Christopher & Connealy, Nathan & Logan, Matthew William, 2024. "“Get rich quick,” scheme or script? The effect of cryptoculture on the susceptibility of fraud victimization among cryptocurrency purchasers," Journal of Criminal Justice, Elsevier, vol. 94(C).
    2. Kusterer, David & Sliwka, Dirk, 2022. "Social Preferences and Rating Biases in Subjective Performance Evaluations," IZA Discussion Papers 15496, Institute of Labor Economics (IZA).
    3. Chapkovski, Philipp, 2023. "Conducting interactive experiments on Toloka," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    4. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    5. Florian Teschner & Henner Gimpel, 2018. "Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments," Group Decision and Negotiation, Springer, vol. 27(2), pages 197-214, April.
    6. Kazakova, E. & Sandomirskaia, M. & Suvorov, A. & Khazhgerieva, A. & Shavshin, R., 2023. "Platforms, online labor markets, and crowdsourcing. Part 2. Crowdsourcing," Journal of the New Economic Association, New Economic Association, vol. 61(4), pages 128-144.
    7. R. Gordon Rinderknecht, 2019. "Effects of Participant Displeasure on the Social-Psychological Study of Power on Amazon’s Mechanical Turk," SAGE Open, , vol. 9(3), pages 21582440198, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schwaiger, Rene & Hueber, Laura, 2021. "Do MTurkers exhibit myopic loss aversion?," Economics Letters, Elsevier, vol. 209(C).
    2. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    3. Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    4. repec:plo:pone00:0198835 is not listed on IDEAS
    5. Blaine G. Robbins, 2017. "Status, identity, and ability in the formation of trust," Rationality and Society, , vol. 29(4), pages 408-448, November.
    6. Austin M Strange & Ryan D Enos & Mark Hill & Amy Lakeman, 2019. "Online volunteer laboratories for human subjects research," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-13, August.
    7. Shari De Baets & Dilek Önkal & Wasim Ahmed, 2022. "Do Risky Scenarios Affect Forecasts of Savings and Expenses?," Forecasting, MDPI, vol. 4(1), pages 1-28, February.
    8. Beshai, Shadi & Mishra, Sandeep & Meadows, Tyler J.S. & Parmar, Priya & Huang, Vivian, 2017. "Minding the gap: Subjective relative deprivation and depressive symptoms," Social Science & Medicine, Elsevier, vol. 173(C), pages 18-25.
    9. Jurásek Miroslav & Potocký Tomislav, 2020. "How to Improve Communication within an Organization? The Relationship between Cultural Intelligence and Language Competence," Journal of Intercultural Management, Sciendo, vol. 12(2), pages 53-81, June.
    10. Sheehan, Ben & Jin, Hyun Seung & Gottlieb, Udo, 2020. "Customer service chatbots: Anthropomorphism and adoption," Journal of Business Research, Elsevier, vol. 115(C), pages 14-24.
    11. Abigail B. Schneider & Bridget Leonard, 2022. "From anxiety to control: Mask‐wearing, perceived marketplace influence, and emotional well‐being during the COVID‐19 pandemic," Journal of Consumer Affairs, Wiley Blackwell, vol. 56(1), pages 97-119, March.
    12. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    13. José M. Ramírez-Hurtado & Alfredo G. Hernández-Díaz & Ana D. López-Sánchez & Víctor E. Pérez-León, 2021. "Measuring Online Teaching Service Quality in Higher Education in the COVID-19 Environment," IJERPH, MDPI, vol. 18(5), pages 1-14, March.
    14. Dwayne D. Gremler & Yves Vaerenbergh & Elisabeth C. Brüggen & Kevin P. Gwinner, 2020. "Understanding and managing customer relational benefits in services: a meta-analysis," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 565-583, May.
    15. Aysun Ata Aktürk & Hasibe Özlen Demircan, 2018. "Development of Preschool Children Sibling Rivalry Scale (PSRS)," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 11(1), pages 117-136, February.
    16. Eugene Y. Chan & Jack Lin, 2022. "Political ideology and psychological reactance: how serious should climate change be?," Climatic Change, Springer, vol. 172(1), pages 1-22, May.
    17. Joana César Machado & Beatriz Fonseca & Carla Martins, 2021. "Brand logo and brand gender: examining the effects of natural logo designs and color on brand gender perceptions and affect," Journal of Brand Management, Palgrave Macmillan, vol. 28(2), pages 152-170, March.
    18. Lucinda Austin & Barbara Miller Gaither, 2019. "Redefining fit: examining CSR company-issue fit in stigmatized industries," Journal of Brand Management, Palgrave Macmillan, vol. 26(1), pages 9-20, January.
    19. Grigorios, Lamprinakos & Magrizos, Solon & Kostopoulos, Ioannis & Drossos, Dimitrios & Santos, David, 2022. "Overt and covert customer data collection in online personalized advertising: The role of user emotions," Journal of Business Research, Elsevier, vol. 141(C), pages 308-320.
    20. Roman Lukyanenko & Andrea Wiggins & Holly K. Rosser, 0. "Citizen Science: An Information Quality Research Frontier," Information Systems Frontiers, Springer, vol. 0, pages 1-23.
    21. Rašković, Matevž & Ding, Zhonghui & Škare, Vatroslav & Ozretić Došen, Đurđana & Žabkar, Vesna, 2016. "Comparing consumer innovativeness and ethnocentrism of young-adult consumers," Journal of Business Research, Elsevier, vol. 69(9), pages 3682-3686.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:7:y:2017:i:2:p:2158244017712774. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.