IDEAS home Printed from https://ideas.repec.org/a/eee/ijrema/v39y2022i1p117-133.html
   My bibliography  Save this article

Collecting samples from online services: How to use screeners to improve data quality

Author

Listed:
  • Arndt, Aaron D.
  • Ford, John B.
  • Babin, Barry J.
  • Luong, Vinh

Abstract

Increasingly, marketing and consumer researchers rely on online data collection services. While actively-managed data collection services directly assist with the sampling process, minimally-managed data collection services, such as Amazon’s Mechanical Turk (MTurk), leave researchers solely responsible for recruiting, screening, cleaning, and evaluating responses. The research reported here proposes a 2 × 2 framework based on sampling goal and methodology for screening and evaluating the quality of online samples. By sampling goals, screeners can be categorized as selection, which involves matching the sample with the targeted population; or as accuracy, which involves ensuring that participants are appropriately attentive. By methodology, screeners can be categorized as direct, which screens individual responses; and as statistical, which provides quantitative signals of low quality. Multiple screeners for each of the four categories are compared across three MTurk samples, two actively-managed data collection samples (Qualtrics and Dynata), and a student sample. The results suggest the need for screening in every online sample, particularly for the MTurk samples, with the fewest supplier-provided filters. Recommendations are provided for researchers and journal reviewers that provide greater transparency with respect to sample practices.

Suggested Citation

  • Arndt, Aaron D. & Ford, John B. & Babin, Barry J. & Luong, Vinh, 2022. "Collecting samples from online services: How to use screeners to improve data quality," International Journal of Research in Marketing, Elsevier, vol. 39(1), pages 117-133.
    • Handle: RePEc:eee:ijrema:v:39:y:2022:i:1:p:117-133
    DOI: 10.1016/j.ijresmar.2021.05.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167811621000203
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijresmar.2021.05.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Thomas Chesney & Kay Penny, 2013. "The Impact of Repeated Lying on Survey Results," SAGE Open, , vol. 3(1), pages 21582440124, January.
    2. Anna Conte & M. Vittoria Levati & Natalia Montinari, 2019. "Experience in public goods experiments," Theory and Decision, Springer, vol. 86(1), pages 65-93, February.
    3. Joseph K. Goodman & Gabriele Paolacci, 2017. "Crowdsourcing Consumer Research," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 44(1), pages 196-210.
    4. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    5. repec:oup:jconrs:v:47:y:2021:i:5:p:675-697. is not listed on IDEAS
    6. Armstrong, J. Scott & Overton, Terry S., 1977. "Estimating Nonresponse Bias in Mail Surveys," MPRA Paper 81694, University Library of Munich, Germany.
    7. Smith, Scott M. & Roster, Catherine A. & Golden, Linda L. & Albaum, Gerald S., 2016. "A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples," Journal of Business Research, Elsevier, vol. 69(8), pages 3139-3148.
    8. Woo, Sang Eun & Keith, Melissa & Thornton, Meghan A., 2015. "Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations," Industrial and Organizational Psychology, Cambridge University Press, vol. 8(2), pages 171-179, June.
    9. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    10. Kathryn Sharpe Wessling & Joel Huber & Oded Netzer, 2017. "MTurk Character Misrepresentation: Assessment and Solutions," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 44(1), pages 211-230.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jonathan Legare & Ping Yao & Victor S. Y. Lo, 2023. "A case for conducting business-to-business experiments with multi-arm multi-stage adaptive designs," Journal of Marketing Analytics, Palgrave Macmillan, vol. 11(3), pages 490-502, September.
    2. Koorank Beheshti, Mohammadali & Gopinath, Mahesh & Ashouri, Sama & Zal, Saeed, 2023. "Does polarizing personality matter in influencer marketing? Evidence from Instagram," Journal of Business Research, Elsevier, vol. 160(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John Hulland & Jeff Miller, 2018. "“Keep on Turkin’”?," Journal of the Academy of Marketing Science, Springer, vol. 46(5), pages 789-794, September.
    2. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.
    3. Martínez-López, Francisco J. & Li, Yangchun & Su, Wan & Feng, Changyuan, 2019. "To have or have not: Buy buttons on social platforms," Journal of Business Research, Elsevier, vol. 105(C), pages 33-48.
    4. Mahmood, Ammara & Luffarelli, Jonathan & Mukesh, Mudra, 2019. "What's in a logo? The impact of complex visual cues in equity crowdfunding," Journal of Business Venturing, Elsevier, vol. 34(1), pages 41-62.
    5. Austin M Strange & Ryan D Enos & Mark Hill & Amy Lakeman, 2019. "Online volunteer laboratories for human subjects research," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-13, August.
    6. Gobena, Lemessa Bayissa & Van Dijke, Marius, 2017. "Fear and caring: Procedural justice, trust, and collective identification as antecedents of voluntary tax compliance," Journal of Economic Psychology, Elsevier, vol. 62(C), pages 1-16.
    7. Marta Pizzetti & Lucia Gatti & Peter Seele, 2021. "Firms Talk, Suppliers Walk: Analyzing the Locus of Greenwashing in the Blame Game and Introducing ‘Vicarious Greenwashing’," Journal of Business Ethics, Springer, vol. 170(1), pages 21-38, April.
    8. Humeyra Dogru-Dastan & Svetlana Stepchenkova & Andrei P. Kirilenko, 2022. "Responding to Visitor Density Pre and Post COVID-19 Outbreak: The Impact of Personality Type on Perceived Crowdedness, Feeling of Being Comfortable, and Anticipated Experience," Sustainability, MDPI, vol. 14(7), pages 1-22, March.
    9. Acemoglu, Daron & Mühlbach, Nicolaj Søndergaard & Scott, Andrew J., 2022. "The rise of age-friendly jobs," The Journal of the Economics of Ageing, Elsevier, vol. 23(C).
    10. Huseyn Abdulla & James D. Abbey & Michael Ketzenberg, 2022. "How consumers value retailer's return policy leniency levers: An empirical investigation," Production and Operations Management, Production and Operations Management Society, vol. 31(4), pages 1719-1733, April.
    11. Jiachun Lu & Lutz Kaufmann & Craig R. Carter, 2021. "How Informal Exchanges Impact Formal Sourcing Collaboration (and What Supply Managers Can Do about It)," Journal of Supply Chain Management, Institute for Supply Management, vol. 57(4), pages 26-62, October.
    12. David Schneider & Johannes Klumpe & Martin Adam & Alexander Benlian, 2020. "Nudging users into digital service solutions," Electronic Markets, Springer;IIM University of St. Gallen, vol. 30(4), pages 863-881, December.
    13. Grégoire, Denis A. & Binder, Julia K. & Rauch, Andreas, 2019. "Navigating the validity tradeoffs of entrepreneurship research experiments: A systematic review and best-practice suggestions," Journal of Business Venturing, Elsevier, vol. 34(2), pages 284-310.
    14. Li Yan & Hean Tat Keh & Xiaoyu Wang, 2021. "Powering Sustainable Consumption: The Roles of Green Consumption Values and Power Distance Belief," Journal of Business Ethics, Springer, vol. 169(3), pages 499-516, March.
    15. Samppa Suoniemi & Lars Meyer-Waarden & Andreas Munzel & Alex Ricardo Zablah & Detmar Straub, 2020. "Big data and firm performance: The roles of market-directed capabilities and business strategy," Post-Print hal-02957479, HAL.
    16. Stephanie Anne Nicole Bedard & Carri Reisdorf Tolmie, 2018. "Millennials' green consumption behaviour: Exploring the role of social media," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 25(6), pages 1388-1396, November.
    17. Nicolas Jacquemet & Alexander James & Stéphane Luchini & James Murphy & Jason F. Shogren, 2019. "Lying and Shirking Under Oath," Working Papers 19-19, Chapman University, Economic Science Institute.
      • Nicolas Jacquemet & Alexander James & Stéphane Luchini & James J. Murphy & Jason F. Shogren, 2019. "Lying and Shirking Under Oath," Working Papers 2019-02, University of Alaska Anchorage, Department of Economics.
    18. Tom Meyvis & Stijn M J Van Osselaer & Dahl DarrenEditor & Eileen FischerEditor & Gita JoharEditor & Vicki MorwitzEditor, 2018. "Increasing the Power of Your Study by Increasing the Effect Size," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 44(5), pages 1157-1173.
    19. Orth, Ulrich R. & Hoffmann, Stefan & Nickel, Kristina, 2019. "Moral decoupling feels good and makes buying counterfeits easy," Journal of Business Research, Elsevier, vol. 98(C), pages 117-125.
    20. Kuchmaner, Christina A. & Wiggins, Jennifer & Grimm, Pamela E., 2019. "The Role of Network Embeddedness and Psychological Ownership in Consumer Responses to Brand Transgressions," Journal of Interactive Marketing, Elsevier, vol. 47(C), pages 129-143.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ijrema:v:39:y:2022:i:1:p:117-133. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/international-journal-of-research-in-marketing/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.