IDEAS home Printed from https://ideas.repec.org/a/eee/ijrema/v39y2022i1p117-133.html
   My bibliography  Save this article

Collecting samples from online services: How to use screeners to improve data quality

Author

Listed:
  • Arndt, Aaron D.
  • Ford, John B.
  • Babin, Barry J.
  • Luong, Vinh

Abstract

Increasingly, marketing and consumer researchers rely on online data collection services. While actively-managed data collection services directly assist with the sampling process, minimally-managed data collection services, such as Amazon’s Mechanical Turk (MTurk), leave researchers solely responsible for recruiting, screening, cleaning, and evaluating responses. The research reported here proposes a 2 × 2 framework based on sampling goal and methodology for screening and evaluating the quality of online samples. By sampling goals, screeners can be categorized as selection, which involves matching the sample with the targeted population; or as accuracy, which involves ensuring that participants are appropriately attentive. By methodology, screeners can be categorized as direct, which screens individual responses; and as statistical, which provides quantitative signals of low quality. Multiple screeners for each of the four categories are compared across three MTurk samples, two actively-managed data collection samples (Qualtrics and Dynata), and a student sample. The results suggest the need for screening in every online sample, particularly for the MTurk samples, with the fewest supplier-provided filters. Recommendations are provided for researchers and journal reviewers that provide greater transparency with respect to sample practices.

Suggested Citation

  • Arndt, Aaron D. & Ford, John B. & Babin, Barry J. & Luong, Vinh, 2022. "Collecting samples from online services: How to use screeners to improve data quality," International Journal of Research in Marketing, Elsevier, vol. 39(1), pages 117-133.
    • Handle: RePEc:eee:ijrema:v:39:y:2022:i:1:p:117-133
    DOI: 10.1016/j.ijresmar.2021.05.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167811621000203
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijresmar.2021.05.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Thomas Chesney & Kay Penny, 2013. "The Impact of Repeated Lying on Survey Results," SAGE Open, , vol. 3(1), pages 21582440124, January.
    2. Smith, Scott M. & Roster, Catherine A. & Golden, Linda L. & Albaum, Gerald S., 2016. "A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples," Journal of Business Research, Elsevier, vol. 69(8), pages 3139-3148.
    3. Anna Conte & M. Vittoria Levati & Natalia Montinari, 2019. "Experience in public goods experiments," Theory and Decision, Springer, vol. 86(1), pages 65-93, February.
    4. Joseph K. Goodman & Gabriele Paolacci, 2017. "Crowdsourcing Consumer Research," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 44(1), pages 196-210.
    5. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    6. repec:oup:jconrs:v:47:y:2021:i:5:p:675-697. is not listed on IDEAS
    7. Woo, Sang Eun & Keith, Melissa & Thornton, Meghan A., 2015. "Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations," Industrial and Organizational Psychology, Cambridge University Press, vol. 8(2), pages 171-179, June.
    8. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    9. Armstrong, J. Scott & Overton, Terry S., 1977. "Estimating Nonresponse Bias in Mail Surveys," MPRA Paper 81694, University Library of Munich, Germany.
    10. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    11. Kathryn Sharpe Wessling & Joel Huber & Oded Netzer, 2017. "MTurk Character Misrepresentation: Assessment and Solutions," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 44(1), pages 211-230.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Danatzis, Ilias & Möller-Herm, Jana & Herm, Steffen, 2024. "Curbing customer-to-customer misbehavior contagion in the sharing economy," Journal of Business Research, Elsevier, vol. 173(C).
    2. Jonathan Legare & Ping Yao & Victor S. Y. Lo, 2023. "A case for conducting business-to-business experiments with multi-arm multi-stage adaptive designs," Journal of Marketing Analytics, Palgrave Macmillan, vol. 11(3), pages 490-502, September.
    3. Agag, Gomaa & Shehawy, Yasser Moustafa & Almoraish, Ahmed & Eid, Riyad & Chaib Lababdi, Houyem & Gherissi Labben, Thouraya & Abdo, Said Shabban, 2024. "Understanding the relationship between marketing analytics, customer agility, and customer satisfaction: A longitudinal perspective," Journal of Retailing and Consumer Services, Elsevier, vol. 77(C).
    4. Koorank Beheshti, Mohammadali & Gopinath, Mahesh & Ashouri, Sama & Zal, Saeed, 2023. "Does polarizing personality matter in influencer marketing? Evidence from Instagram," Journal of Business Research, Elsevier, vol. 160(C).
    5. Mangus, Stephanie M. & Shi, Huanhuan & Folse, Judith Anne Garretson & Jones, Eli & Sridhar, Shrihari, 2024. "Communicating with B2B buyers after “Dropping the Ball”: Using digital and non-digital communication formats to recover from salesperson transgressions," International Journal of Research in Marketing, Elsevier, vol. 41(2), pages 194-219.
    6. Lee, Zoe & Faridah Syed Alwi, Sharifah & Gambetti, Rossella, 2024. "The thousand faces of beauty: How credible storytelling unlocks disability representation in inclusive luxury fashion branding," Journal of Business Research, Elsevier, vol. 181(C).
    7. Subarna Nandy & Neena Sondhi & Himanshu Joshi, 2024. "Toward a measure of brand pride: scale development and validation," Journal of Brand Management, Palgrave Macmillan, vol. 31(4), pages 430-448, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John Hulland & Jeff Miller, 2018. "“Keep on Turkin’”?," Journal of the Academy of Marketing Science, Springer, vol. 46(5), pages 789-794, September.
    2. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.
    3. Mahmood, Ammara & Luffarelli, Jonathan & Mukesh, Mudra, 2019. "What's in a logo? The impact of complex visual cues in equity crowdfunding," Journal of Business Venturing, Elsevier, vol. 34(1), pages 41-62.
    4. Haas, Nicholas & Hassan, Mazen & Mansour, Sarah & Morton, Rebecca B., 2021. "Polarizing information and support for reform," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 883-901.
    5. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    6. Atalay, Kadir & Bakhtiar, Fayzan & Cheung, Stephen & Slonim, Robert, 2014. "Savings and prize-linked savings accounts," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 86-106.
    7. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.
    8. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    9. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    10. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    11. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    12. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    13. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.
    14. Orazi, Davide C. & Johnston, Allen C., 2020. "Running field experiments using Facebook split test," Journal of Business Research, Elsevier, vol. 118(C), pages 189-198.
    15. Henkens, Bieke & Verleye, Katrien & Larivière, Bart, 2021. "The smarter, the better?! Customer well-being, engagement, and perceptions in smart service systems," International Journal of Research in Marketing, Elsevier, vol. 38(2), pages 425-447.
    16. Carpiano, Richard M. & Fitz, Nicholas S., 2017. "Public attitudes toward child undervaccination: A randomized experiment on evaluations, stigmatizing orientations, and support for policies," Social Science & Medicine, Elsevier, vol. 185(C), pages 127-136.
    17. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
    18. Martínez-López, Francisco J. & Li, Yangchun & Su, Wan & Feng, Changyuan, 2019. "To have or have not: Buy buttons on social platforms," Journal of Business Research, Elsevier, vol. 105(C), pages 33-48.
    19. Thomas W. H. Ng & Lorenzo Lucianetti & Dennis Y. Hsu & Frederick H. K. Yim & Kelly L. Sorensen, 2021. "You Speak, I Speak: The Social‐Cognitive Mechanisms of Voice Contagion," Journal of Management Studies, Wiley Blackwell, vol. 58(6), pages 1569-1608, September.
    20. Murnieks, Charles Y. & Arthurs, Jonathan D. & Cardon, Melissa S. & Farah, Nusrat & Stornelli, Jason & Michael Haynie, J., 2020. "Close your eyes or open your mind: Effects of sleep and mindfulness exercises on entrepreneurs' exhaustion," Journal of Business Venturing, Elsevier, vol. 35(2).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ijrema:v:39:y:2022:i:1:p:117-133. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/international-journal-of-research-in-marketing/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.