IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0279720.html
   My bibliography  Save this article

Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

Author

Listed:
  • Benjamin D Douglas
  • Patrick J Ewell
  • Markus Brauer

Abstract

With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher’s question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.

Suggested Citation

  • Benjamin D Douglas & Patrick J Ewell & Markus Brauer, 2023. "Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA," PLOS ONE, Public Library of Science, vol. 18(3), pages 1-17, March.
    • Handle: RePEc:plo:pone00:0279720
    DOI: 10.1371/journal.pone.0279720
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0279720
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0279720&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0279720?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Smith, Scott M. & Roster, Catherine A. & Golden, Linda L. & Albaum, Gerald S., 2016. "A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples," Journal of Business Research, Elsevier, vol. 69(8), pages 3139-3148.
    2. repec:plo:pone00:0157732 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pan, Yu & Pereira, Marco Henriques & Gomez-Gonzalez, Carlos & Dietl, Helmut M., 2025. "The superstar effect on perceived performance in professional football: An online experiment," Journal of Economic Psychology, Elsevier, vol. 106(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Anastasia Stathopoulou & Tommy Kweku Quansah & George Balabanis, 2022. "The Blinding Effects of Team Identification on Sports Corruption: Cross-Cultural Evidence from Sub-Saharan African Countries," Journal of Business Ethics, Springer, vol. 179(2), pages 511-529, August.
    2. Shir-Way Siew & Michael S. Minor & Reto Felix, 2018. "The influence of perceived strength of brand origin on willingness to pay more for luxury goods," Journal of Brand Management, Palgrave Macmillan, vol. 25(6), pages 591-605, November.
    3. Alina Ferecatu & Arnaud Bruyn & Prithwiraj Mukherjee, 2024. "Silently killing your panelists one email at a time: The true cost of email solicitations," Journal of the Academy of Marketing Science, Springer, vol. 52(4), pages 1216-1239, July.
    4. Kazi, Oishi & Miller, Steven R. & Malone, Trey & Wolf, Christopher A., 2022. "The Changing Role of Fat Perceptions in Fluid Milk Labeling: Would the Dairy Industry Sell More if 2% Milk Was Called “98% Fat Free”?," Journal of Food Distribution Research, Food Distribution Research Society, vol. 53(2), July.
    5. van Esch, Patrick & Black, J. Stewart, 2019. "Factors that influence new generation candidates to engage with and complete digital, AI-enabled recruiting," Business Horizons, Elsevier, vol. 62(6), pages 729-739.
    6. Mwesiumo, Deodat & Halpern, Nigel & Budd, Thomas & Suau-Sanchez, Pere & Bråthen, Svein, 2021. "An exploratory and confirmatory composite analysis of a scale for measuring privacy concerns," Journal of Business Research, Elsevier, vol. 136(C), pages 63-75.
    7. Vieira, Valter Afonso & Rafael, Diego Nogueira & Agnihotri, Raj, 2022. "Augmented reality generalizations: A meta-analytical review on consumer-related outcomes and the mediating role of hedonic and utilitarian values," Journal of Business Research, Elsevier, vol. 151(C), pages 170-184.
    8. Orazi, Davide C. & Johnston, Allen C., 2020. "Running field experiments using Facebook split test," Journal of Business Research, Elsevier, vol. 118(C), pages 189-198.
    9. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.
    10. Rudolph, Lukas & Freitag, Markus & Thurner, Paul, 2021. "The Comparative Legitimacy of Arms Exports - A Conjoint Experiment in Germany and France," SocArXiv r73pv, Center for Open Science.
    11. Yang, Xisi & Thøgersen, John, 2022. "When people are green and greedy: A new perspective of recycling rewards and crowding-out in Germany, the USA and China," Journal of Business Research, Elsevier, vol. 144(C), pages 217-235.
    12. repec:agr:journl:v:1(618):y:2019:i:1(618):p:5-22 is not listed on IDEAS
    13. Alberto Michele Felicetti & Antonio Palmiro Volpentesta & Roberto Linzalone & Salvatore Ammirato, 2023. "Information Behaviour of Food Consumers: A Systematic Literature Review and a Future Research Agenda," Sustainability, MDPI, vol. 15(4), pages 1-20, February.
    14. Lehmann, Nico & Sloot, Daniel & Ardone, Armin & Fichtner, Wolf, 2021. "The limited potential of regional electricity marketing – Results from two discrete choice experiments in Germany," Energy Economics, Elsevier, vol. 100(C).
    15. Erika Austhof & Heidi E. Brown, 2022. "Global Warming’s Six MTurks: A Secondary Analysis of a US-Based Online Crowdsourcing Market," IJERPH, MDPI, vol. 19(14), pages 1-9, July.
    16. Moon, Sangkil & Kim, Moon-Yong & Bergey, Paul K., 2019. "Estimating deception in consumer reviews based on extreme terms: Comparison analysis of open vs. closed hotel reservation platforms," Journal of Business Research, Elsevier, vol. 102(C), pages 83-96.
    17. Ina Garnefeld & Andreas Eggert & Markus Husemann-Kopetzky & Eva Böhm, 2019. "Exploring the link between payment schemes and customer fraud: a mental accounting perspective," Journal of the Academy of Marketing Science, Springer, vol. 47(4), pages 595-616, July.
    18. Waqar Nadeem & Mari Juntunen & Nick Hajli & Mina Tajvidi, 2021. "The Role of Ethical Perceptions in Consumers’ Participation and Value Co-creation on Sharing Economy Platforms," Journal of Business Ethics, Springer, vol. 169(3), pages 421-441, March.
    19. Arndt, Aaron D. & Ford, John B. & Babin, Barry J. & Luong, Vinh, 2022. "Collecting samples from online services: How to use screeners to improve data quality," International Journal of Research in Marketing, Elsevier, vol. 39(1), pages 117-133.
    20. Humeyra Dogru-Dastan & Svetlana Stepchenkova & Andrei P. Kirilenko, 2022. "Responding to Visitor Density Pre and Post COVID-19 Outbreak: The Impact of Personality Type on Perceived Crowdedness, Feeling of Being Comfortable, and Anticipated Experience," Sustainability, MDPI, vol. 14(7), pages 1-22, March.
    21. Sung, Billy & Crawford, Robert & Teah, Min & Stankovic, Michelle & Phau, Ian, 2020. "The “timber box†effect for premium wines," Journal of Retailing and Consumer Services, Elsevier, vol. 54(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0279720. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.