IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0279720.html
   My bibliography  Save this article

Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

Author

Listed:
  • Benjamin D Douglas
  • Patrick J Ewell
  • Markus Brauer

Abstract

With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher’s question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.

Suggested Citation

  • Benjamin D Douglas & Patrick J Ewell & Markus Brauer, 2023. "Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA," PLOS ONE, Public Library of Science, vol. 18(3), pages 1-17, March.
    • Handle: RePEc:plo:pone00:0279720
    DOI: 10.1371/journal.pone.0279720
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0279720
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0279720&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0279720?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Roulin, Nicolas, 2015. "Don't Throw the Baby Out With the Bathwater: Comparing Data Quality of Crowdsourcing, Online Panels, and Student Samples," Industrial and Organizational Psychology, Cambridge University Press, vol. 8(2), pages 190-196, June.
    2. Smith, Scott M. & Roster, Catherine A. & Golden, Linda L. & Albaum, Gerald S., 2016. "A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples," Journal of Business Research, Elsevier, vol. 69(8), pages 3139-3148.
    3. repec:plo:pone00:0157732 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kit S. Double & Hester Xiao & Rebecca T. Pinkus & Sarah A. Walker & Carolyn MacCann, 2025. "You Feel Better When Your Partner is Emotionally Intelligent: Self-Rated Emotional Intelligence Shows Partner Effects on Subjective Wellbeing," Journal of Happiness Studies, Springer, vol. 26(5), pages 1-17, June.
    2. Chih, Wen-Hai & Wang, Kai-Yu & Banda, Hope Wilfred, 2025. "Chatbots at the frontline: Unveiling antecedents of customers’ willingness to accept chatbot intervention in service recovery," Journal of Retailing and Consumer Services, Elsevier, vol. 84(C).
    3. Pan, Yu & Pereira, Marco Henriques & Gomez-Gonzalez, Carlos & Dietl, Helmut M., 2025. "The superstar effect on perceived performance in professional football: An online experiment," Journal of Economic Psychology, Elsevier, vol. 106(C).
    4. Jedicke, Eva Maria & Arnold, Christian & Lindenmeier, Jörg, 2025. "The dynamics of consumer boycott intention: Examining the roles of moral reasoning, cognitive dissonance, and self-congruence," Journal of Business Research, Elsevier, vol. 189(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Capri Ka Po Kong & Miu Chung Yan & Sean Lauer & Shao Hua Zhan, 2023. "Immigrant Identifications and ICT Use: A Survey Study of Chinese and South Asian Immigrants in Canada," Journal of International Migration and Integration, Springer, vol. 24(2), pages 885-910, June.
    2. Kim, Kyoung Yong & Patel, Pankaj C. & Devaraj, Srikant, 2025. "Sorting, incentive, and investment effects of employee stock ownership," Journal of Business Research, Elsevier, vol. 191(C).
    3. Anastasia Stathopoulou & Tommy Kweku Quansah & George Balabanis, 2022. "The Blinding Effects of Team Identification on Sports Corruption: Cross-Cultural Evidence from Sub-Saharan African Countries," Journal of Business Ethics, Springer, vol. 179(2), pages 511-529, August.
    4. Shir-Way Siew & Michael S. Minor & Reto Felix, 2018. "The influence of perceived strength of brand origin on willingness to pay more for luxury goods," Journal of Brand Management, Palgrave Macmillan, vol. 25(6), pages 591-605, November.
    5. Alina Ferecatu & Arnaud Bruyn & Prithwiraj Mukherjee, 2024. "Silently killing your panelists one email at a time: The true cost of email solicitations," Journal of the Academy of Marketing Science, Springer, vol. 52(4), pages 1216-1239, July.
    6. Kazi, Oishi & Miller, Steven R. & Malone, Trey & Wolf, Christopher A., 2022. "The Changing Role of Fat Perceptions in Fluid Milk Labeling: Would the Dairy Industry Sell More if 2% Milk Was Called “98% Fat Free”?," Journal of Food Distribution Research, Food Distribution Research Society, vol. 53(2), July.
    7. Kaitlynn Sandstrom‐Mistry & Frank Lupi & Hyunjung Kim & Joseph A. Herriges, 2023. "Comparing water quality valuation across probability and non‐probability samples," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 45(2), pages 744-761, June.
    8. van Esch, Patrick & Black, J. Stewart, 2019. "Factors that influence new generation candidates to engage with and complete digital, AI-enabled recruiting," Business Horizons, Elsevier, vol. 62(6), pages 729-739.
    9. Mwesiumo, Deodat & Halpern, Nigel & Budd, Thomas & Suau-Sanchez, Pere & Bråthen, Svein, 2021. "An exploratory and confirmatory composite analysis of a scale for measuring privacy concerns," Journal of Business Research, Elsevier, vol. 136(C), pages 63-75.
    10. Vieira, Valter Afonso & Rafael, Diego Nogueira & Agnihotri, Raj, 2022. "Augmented reality generalizations: A meta-analytical review on consumer-related outcomes and the mediating role of hedonic and utilitarian values," Journal of Business Research, Elsevier, vol. 151(C), pages 170-184.
    11. Orazi, Davide C. & Johnston, Allen C., 2020. "Running field experiments using Facebook split test," Journal of Business Research, Elsevier, vol. 118(C), pages 189-198.
    12. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.
    13. Nitika Sharma & Arminda Paço & Raysa Geaquinto Rocha & Maria Palazzo & Alfonso Siano, 2024. "Examining a theoretical model of eco‐anxiety on consumers' intentions towards green products," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 31(3), pages 1868-1885, May.
    14. Rudolph, Lukas & Freitag, Markus & Thurner, Paul, 2021. "The Comparative Legitimacy of Arms Exports - A Conjoint Experiment in Germany and France," SocArXiv r73pv, Center for Open Science.
    15. Yang, Xisi & Thøgersen, John, 2022. "When people are green and greedy: A new perspective of recycling rewards and crowding-out in Germany, the USA and China," Journal of Business Research, Elsevier, vol. 144(C), pages 217-235.
    16. Adam P. Mayer & E. Keith Smith, 2023. "Multidimensional partisanship shapes climate policy support and behaviours," Nature Climate Change, Nature, vol. 13(1), pages 32-39, January.
    17. repec:agr:journl:v:1(618):y:2019:i:1(618):p:5-22 is not listed on IDEAS
    18. Alberto Michele Felicetti & Antonio Palmiro Volpentesta & Roberto Linzalone & Salvatore Ammirato, 2023. "Information Behaviour of Food Consumers: A Systematic Literature Review and a Future Research Agenda," Sustainability, MDPI, vol. 15(4), pages 1-20, February.
    19. Lehmann, Nico & Sloot, Daniel & Ardone, Armin & Fichtner, Wolf, 2021. "The limited potential of regional electricity marketing – Results from two discrete choice experiments in Germany," Energy Economics, Elsevier, vol. 100(C).
    20. repec:osf:osfxxx:2hez9_v1 is not listed on IDEAS
    21. Erika Austhof & Heidi E. Brown, 2022. "Global Warming’s Six MTurks: A Secondary Analysis of a US-Based Online Crowdsourcing Market," IJERPH, MDPI, vol. 19(14), pages 1-9, July.
    22. Moon, Sangkil & Kim, Moon-Yong & Bergey, Paul K., 2019. "Estimating deception in consumer reviews based on extreme terms: Comparison analysis of open vs. closed hotel reservation platforms," Journal of Business Research, Elsevier, vol. 102(C), pages 83-96.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0279720. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.