IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v7y2017i2p2158244017712774.html
   My bibliography  Save this article

Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection

Author

Listed:
  • Logan S. Casey
  • Jesse Chandler
  • Adam Seth Levine
  • Andrew Proctor
  • Dara Z. Strolovitch

Abstract

The online labor market Amazon Mechanical Turk (MTurk) is an increasingly popular source of respondents for social science research. A growing body of research has examined the demographic composition of MTurk workers as compared with that of other populations. While these comparisons have revealed the ways in which MTurk workers are and are not representative of the general population, variations among samples drawn from MTurk have received less attention. This article focuses on whether MTurk sample composition varies as a function of time. Specifically, we examine whether demographic characteristics vary by (a) time of day, (b) day of week, and serial position (i.e., earlier or later in data collection), both (c) across the entire data collection and (d) within specific batches. We find that day of week differences are minimal, but that time of day and serial position are associated with small but important variations in demographic composition. This demonstrates that MTurk samples cannot be presumed identical across different studies, potentially affecting reliability, validity, and efforts to reproduce findings.

Suggested Citation

  • Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    • Handle: RePEc:sae:sagope:v:7:y:2017:i:2:p:2158244017712774
    DOI: 10.1177/2158244017712774
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244017712774
    Download Restriction: no

    References listed on IDEAS

    as
    1. R Chris Fraley & Simine Vazire, 2014. "The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power," PLOS ONE, Public Library of Science, vol. 9(10), pages 1-12, October.
    2. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    3. Robert Peterson & Dwight Merunka, 2014. "Convenience samples of college students and research reproducibility," Post-Print hal-01822317, HAL.
    4. Neil Stewart & Christoph Ungemach & Adam J. L. Harris & Daniel M. Bartels & Ben R. Newell & Gabriele Paolacci & Jesse Chandler, 2015. "The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(5), pages 479-491, September.
    5. Krupnikov, Yanna & Levine, Adam Seth, 2014. "Cross-Sample Comparisons and External Validity," Journal of Experimental Political Science, Cambridge University Press, vol. 1(1), pages 59-80, April.
    6. Peterson, Robert A. & Merunka, Dwight R., 2014. "Convenience samples of college students and research reproducibility," Journal of Business Research, Elsevier, vol. 67(5), pages 1035-1041.
    7. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    8. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. R. Gordon Rinderknecht, 2019. "Effects of Participant Displeasure on the Social-Psychological Study of Power on Amazon’s Mechanical Turk," SAGE Open, , vol. 9(3), pages 21582440198, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    2. Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    3. Kelly Walters & Dimitri A Christakis & Davene R Wright, 2018. "Are Mechanical Turk worker samples representative of health status and health behaviors in the U.S.?," PLOS ONE, Public Library of Science, vol. 13(6), pages 1-10, June.
    4. Mireia Ercilla-Montserrat & David Sanjuan-Delmás & Esther Sanyé-Mengual & Laura Calvet-Mir & Karla Banderas & Joan Rieradevall & Xavier Gabarrell, 2019. "Analysis of the consumer’s perception of urban food products from a soilless system in rooftop greenhouses: a case study from the Mediterranean area of Barcelona (Spain)," Agriculture and Human Values, Springer;The Agriculture, Food, & Human Values Society (AFHVS), vol. 36(3), pages 375-393, September.
    5. Dwayne D. Gremler & Yves Vaerenbergh & Elisabeth C. Brüggen & Kevin P. Gwinner, 2020. "Understanding and managing customer relational benefits in services: a meta-analysis," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 565-583, May.
    6. Lucinda Austin & Barbara Miller Gaither, 2019. "Redefining fit: examining CSR company-issue fit in stigmatized industries," Journal of Brand Management, Palgrave Macmillan, vol. 26(1), pages 9-20, January.
    7. Teng, Weichen & Su, Yaohua & Liao, Tien-Tien & Wei, Cang-Liang, 2020. "An exploration of celebrity business ventures and their appeal to fans and non-fans," Journal of Retailing and Consumer Services, Elsevier, vol. 54(C).
    8. Nadeem, Waqar & Al-Imamy, Saifeddin, 2020. "Do ethics drive value co-creation on digital sharing economy platforms?," Journal of Retailing and Consumer Services, Elsevier, vol. 55(C).
    9. Nadeem, Waqar & Juntunen, Mari & Shirazi, Farid & Hajli, Nick, 2020. "Consumers’ value co-creation in sharing economy: The role of social support, consumers’ ethical perceptions and relationship quality," Technological Forecasting and Social Change, Elsevier, vol. 151(C).
    10. Dunja Meštrović & Nina J. Zugic, 2018. "Students’ Perceptions of Service Quality, Satisfaction, and Word-Of-Mouth: Scale Adaptation and Validation on a Sample of ICT and STEM Students," Tržište/Market, Faculty of Economics and Business, University of Zagreb, vol. 30(2), pages 195-211.
    11. Rašković, Matevž & Ding, Zhonghui & Hirose, Morikazu & Žabkar, Vesna & Fam, Kim-Shyan, 2020. "Segmenting young-adult consumers in East Asia and Central and Eastern Europe – The role of consumer ethnocentrism and decision-making styles," Journal of Business Research, Elsevier, vol. 108(C), pages 496-507.
    12. Fagerstrøm, Asle & Eriksson, Niklas & Sigurdsson, Valdimar, 2020. "Investigating the impact of Internet of Things services from a smartphone app on grocery shopping," Journal of Retailing and Consumer Services, Elsevier, vol. 52(C).
    13. David Ronayne & Daniel Sgroi, 2018. "On the motivations for the dual-use of electronic and traditional cigarettes," Applied Economics Letters, Taylor & Francis Journals, vol. 25(12), pages 830-834, July.
    14. Christ, Margaret H. & Vance, Thomas W., 2018. "Cascading controls: The effects of managers’ incentives on subordinate effort to help or harm," Accounting, Organizations and Society, Elsevier, vol. 65(C), pages 20-32.
    15. Dooyoung Choi & Ha Kyung Lee, 2020. "Beneficiary Foci Types and Performance Appeals in Green Advertising," Sustainability, MDPI, Open Access Journal, vol. 12(15), pages 1-15, August.
    16. Mandy Ryan & Nicolas Krucien & Frouke Hermens, 2018. "The eyes have it: Using eye tracking to inform information processing strategies in multi‐attributes choices," Health Economics, John Wiley & Sons, Ltd., vol. 27(4), pages 709-721, April.
    17. D. A. L. Coldwell, 2017. "Custom and Moral Sentiment: Cross-Cultural Aspects of Postgraduate Student Perceptions of Leadership Ethicality," Journal of Business Ethics, Springer, vol. 145(1), pages 201-213, September.
    18. Sebastian Lehmann, 2014. "Toward an Understanding of the BDM: Predictive Validity, Gambling Effects, and Risk Attitude," FEMM Working Papers 150001, Otto-von-Guericke University Magdeburg, Faculty of Economics and Management.
    19. Robert A. Peterson & U. N. Umesh, 2018. "On the significance of statistically insignificant results in consumer behavior experiments," Journal of the Academy of Marketing Science, Springer, vol. 46(1), pages 81-91, January.
    20. Qin, Hong & Peak, Daniel Alan & Prybutok, Victor, 2021. "A virtual market in your pocket: How does mobile augmented reality (MAR) influence consumer decision making?," Journal of Retailing and Consumer Services, Elsevier, vol. 58(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:7:y:2017:i:2:p:2158244017712774. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (SAGE Publications). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.