IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0279720.html
   My bibliography  Save this article

Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

Author

Listed:
  • Benjamin D Douglas
  • Patrick J Ewell
  • Markus Brauer

Abstract

With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher’s question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.

Suggested Citation

  • Benjamin D Douglas & Patrick J Ewell & Markus Brauer, 2023. "Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA," PLOS ONE, Public Library of Science, vol. 18(3), pages 1-17, March.
    • Handle: RePEc:plo:pone00:0279720
    DOI: 10.1371/journal.pone.0279720
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0279720
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0279720&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0279720?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:plo:pone00:0157732 is not listed on IDEAS
    2. Roulin, Nicolas, 2015. "Don't Throw the Baby Out With the Bathwater: Comparing Data Quality of Crowdsourcing, Online Panels, and Student Samples," Industrial and Organizational Psychology, Cambridge University Press, vol. 8(2), pages 190-196, June.
    3. Smith, Scott M. & Roster, Catherine A. & Golden, Linda L. & Albaum, Gerald S., 2016. "A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples," Journal of Business Research, Elsevier, vol. 69(8), pages 3139-3148.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kit S. Double & Hester Xiao & Rebecca T. Pinkus & Sarah A. Walker & Carolyn MacCann, 2025. "You Feel Better When Your Partner is Emotionally Intelligent: Self-Rated Emotional Intelligence Shows Partner Effects on Subjective Wellbeing," Journal of Happiness Studies, Springer, vol. 26(5), pages 1-17, June.
    2. Pan, Yu & Pereira, Marco Henriques & Gomez-Gonzalez, Carlos & Dietl, Helmut M., 2025. "The superstar effect on perceived performance in professional football: An online experiment," Journal of Economic Psychology, Elsevier, vol. 106(C).
    3. Chih, Wen-Hai & Wang, Kai-Yu & Banda, Hope Wilfred, 2025. "Chatbots at the frontline: Unveiling antecedents of customers’ willingness to accept chatbot intervention in service recovery," Journal of Retailing and Consumer Services, Elsevier, vol. 84(C).
    4. Jedicke, Eva Maria & Arnold, Christian & Lindenmeier, Jörg, 2025. "The dynamics of consumer boycott intention: Examining the roles of moral reasoning, cognitive dissonance, and self-congruence," Journal of Business Research, Elsevier, vol. 189(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Anastasia Stathopoulou & Tommy Kweku Quansah & George Balabanis, 2022. "The Blinding Effects of Team Identification on Sports Corruption: Cross-Cultural Evidence from Sub-Saharan African Countries," Journal of Business Ethics, Springer, vol. 179(2), pages 511-529, August.
    2. Shir-Way Siew & Michael S. Minor & Reto Felix, 2018. "The influence of perceived strength of brand origin on willingness to pay more for luxury goods," Journal of Brand Management, Palgrave Macmillan, vol. 25(6), pages 591-605, November.
    3. Alina Ferecatu & Arnaud Bruyn & Prithwiraj Mukherjee, 2024. "Silently killing your panelists one email at a time: The true cost of email solicitations," Journal of the Academy of Marketing Science, Springer, vol. 52(4), pages 1216-1239, July.
    4. Kazi, Oishi & Miller, Steven R. & Malone, Trey & Wolf, Christopher A., 2022. "The Changing Role of Fat Perceptions in Fluid Milk Labeling: Would the Dairy Industry Sell More if 2% Milk Was Called “98% Fat Free”?," Journal of Food Distribution Research, Food Distribution Research Society, vol. 53(2), July.
    5. van Esch, Patrick & Black, J. Stewart, 2019. "Factors that influence new generation candidates to engage with and complete digital, AI-enabled recruiting," Business Horizons, Elsevier, vol. 62(6), pages 729-739.
    6. Mwesiumo, Deodat & Halpern, Nigel & Budd, Thomas & Suau-Sanchez, Pere & Bråthen, Svein, 2021. "An exploratory and confirmatory composite analysis of a scale for measuring privacy concerns," Journal of Business Research, Elsevier, vol. 136(C), pages 63-75.
    7. Vieira, Valter Afonso & Rafael, Diego Nogueira & Agnihotri, Raj, 2022. "Augmented reality generalizations: A meta-analytical review on consumer-related outcomes and the mediating role of hedonic and utilitarian values," Journal of Business Research, Elsevier, vol. 151(C), pages 170-184.
    8. Rudolph, Lukas & Freitag, Markus & Thurner, Paul, 2021. "The Comparative Legitimacy of Arms Exports - A Conjoint Experiment in Germany and France," SocArXiv r73pv, Center for Open Science.
    9. Adam P. Mayer & E. Keith Smith, 2023. "Multidimensional partisanship shapes climate policy support and behaviours," Nature Climate Change, Nature, vol. 13(1), pages 32-39, January.
    10. repec:agr:journl:v:1(618):y:2019:i:1(618):p:5-22 is not listed on IDEAS
    11. Alberto Michele Felicetti & Antonio Palmiro Volpentesta & Roberto Linzalone & Salvatore Ammirato, 2023. "Information Behaviour of Food Consumers: A Systematic Literature Review and a Future Research Agenda," Sustainability, MDPI, vol. 15(4), pages 1-20, February.
    12. repec:osf:osfxxx:2hez9_v1 is not listed on IDEAS
    13. Erika Austhof & Heidi E. Brown, 2022. "Global Warming’s Six MTurks: A Secondary Analysis of a US-Based Online Crowdsourcing Market," IJERPH, MDPI, vol. 19(14), pages 1-9, July.
    14. Ina Garnefeld & Andreas Eggert & Markus Husemann-Kopetzky & Eva Böhm, 2019. "Exploring the link between payment schemes and customer fraud: a mental accounting perspective," Journal of the Academy of Marketing Science, Springer, vol. 47(4), pages 595-616, July.
    15. Arndt, Aaron D. & Ford, John B. & Babin, Barry J. & Luong, Vinh, 2022. "Collecting samples from online services: How to use screeners to improve data quality," International Journal of Research in Marketing, Elsevier, vol. 39(1), pages 117-133.
    16. Saridakis, Charalampos & Katsikeas, Constantine S. & Angelidou, Sofia & Oikonomidou, Maria & Pratikakis, Polyvios, 2023. "Mining Twitter lists to extract brand-related associative information for celebrity endorsement," European Journal of Operational Research, Elsevier, vol. 311(1), pages 316-332.
    17. Pelletier, Mark J. & Horky, Alisha Blakeney & Fox, Alexa K., 2021. "Fexit: The effect of political and promotional communication from friends and family on Facebook exiting intentions," Journal of Business Research, Elsevier, vol. 122(C), pages 321-334.
    18. Monahan, Lisa & Espinosa, Jennifer A. & Langenderfer, Jeff & Ortinau, David J., 2023. "Did you hear our brand is hated? The unexpected upside of hate-acknowledging advertising for polarizing brands," Journal of Business Research, Elsevier, vol. 154(C).
    19. van Esch, Patrick & Cui, Yuanyuan (Gina) & Jain, Shailendra Pratap, 2021. "The effect of political ideology and message frame on donation intent during the COVID-19 pandemic," Journal of Business Research, Elsevier, vol. 125(C), pages 201-213.
    20. Akram, Muhammad S. & Malhotra, Neeru & Goraya, M. Awais Shakir & Shareef, Mahmud A. & Malik, Aneela & Lal, Banita, 2022. "User engagement on global social networks: Examining the roles of perceived brand globalness, identification and global identity," Technological Forecasting and Social Change, Elsevier, vol. 181(C).
    21. Höllig, Christoph E. & Tumasjan, Andranik & Welpe, Isabell M., 2020. "Individualizing gamified systems: The role of trait competitiveness and leaderboard design," Journal of Business Research, Elsevier, vol. 106(C), pages 288-303.
    22. Ang, Ming Yang Avon & Pontes, Nicolas & France, Cassandra, 2024. "Unsustainable burgers? Deploying carbon footprint labels to enhance sustainability perceptions of animal-based food products," Journal of Retailing and Consumer Services, Elsevier, vol. 76(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0279720. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.