IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/cv2bn.html

Methods to Detect Low Quality Data and Its Implication for Psychological Research

Author

Listed:
  • Buchanan, Erin Michelle

    (Harrisburg University of Science and Technology)

  • Scofield, John E.

Abstract

Web-based data collection methods such as Amazon's Mechanical Turk (AMT) are an appealing option to recruit participants quickly and cheaply for psychological research. While concerns regarding data quality have emerged with AMT, several studies have exhibited that data collected via AMT are as reliable as traditional college samples and are often more diverse and representative of noncollege populations. The development of methods to screen for low quality data, however, has been less explored. Omitting participants based on simple screening methods in isolation, such as response time or attention checks may not be adequate identification methods, with an inability to delineate between high or low effort participants. Additionally, problematic survey responses may arise from survey automation techniques such as survey bots or automated form fillers. The current project developed low quality data detection methods while overcoming previous screening limitations. Multiple checks were employed, such as page response times, distribution of survey responses, the number of utilized choices from a given range of scale options, click counts, and manipulation checks. This method was tested on a survey taken with an easily available plug-in survey bot, as well as compared to data collected by human participants providing both high effort and randomized, or low effort, answers. Identified cases can then be used as part of sensitivity analyses to warrant exclusion from further analyses. This algorithm can be a promising tool to identify low quality or automated data via AMT or other online data collection platforms.

Suggested Citation

  • Buchanan, Erin Michelle & Scofield, John E., 2017. "Methods to Detect Low Quality Data and Its Implication for Psychological Research," OSF Preprints cv2bn, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:cv2bn
    DOI: 10.31219/osf.io/cv2bn
    as

    Download full text from publisher

    File URL: https://osf.io/download/59d527089ad5a102b1a1e58b/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/cv2bn?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    2. Joseph Henrich & Steve J. Heine & Ara Norenzayan, 2010. "The Weirdest People in the World?," RatSWD Working Papers 139, German Data Forum (RatSWD).
    3. Rajagopal, 2014. "The Human Factors," Palgrave Macmillan Books, in: Architecting Enterprise, chapter 9, pages 225-249, Palgrave Macmillan.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    2. Lutz, Christoph & Newlands, Gemma, 2018. "Consumer segmentation within the sharing economy: The case of Airbnb," Journal of Business Research, Elsevier, vol. 88(C), pages 187-196.
    3. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
    4. Lars Hornuf & Daniel Vrankar, 2022. "Hourly Wages in Crowdworking: A Meta-Analysis," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(5), pages 553-573, October.
    5. Björn Hedin & Cecilia Katzeff & Elina Eriksson & Daniel Pargman, 2019. "A Systematic Review of Digital Behaviour Change Interventions for More Sustainable Food Consumption," Sustainability, MDPI, vol. 11(9), pages 1-23, May.
    6. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    7. Kadic-Maglajlic, Selma & Arslanagic-Kalajdzic, Maja & Micevski, Milena & Dlacic, Jasmina & Zabkar, Vesna, 2019. "Being engaged is a good thing: Understanding sustainable consumption behavior among young adults," Journal of Business Research, Elsevier, vol. 104(C), pages 644-654.
    8. Maier, Michael F. & Viete, Steffen & Ody, Margard, 2017. "Plattformbasierte Erwerbsarbeit: Stand der empirischen Forschung," IZA Research Reports 81, Institute of Labor Economics (IZA).
    9. Christian Fieseler & Eliane Bucher & Christian Pieter Hoffmann, 2019. "Unfairness by Design? The Perceived Fairness of Digital Labor on Crowdworking Platforms," Journal of Business Ethics, Springer, vol. 156(4), pages 987-1005, June.
    10. Yasmina Okan & Eric R. Stone & Wändi Bruine de Bruin, 2018. "Designing Graphs that Promote Both Risk Understanding and Behavior Change," Risk Analysis, John Wiley & Sons, vol. 38(5), pages 929-946, May.
    11. Sayooran Nagulendra & Julita Vassileva, 2016. "Providing awareness, explanation and control of personalized filtering in a social networking site," Information Systems Frontiers, Springer, vol. 18(1), pages 145-158, February.
    12. Maria Tsourela & Dafni-Maria Nerantzaki, 2020. "An Internet of Things (IoT) Acceptance Model. Assessing Consumer’s Behavior toward IoT Products and Applications," Future Internet, MDPI, vol. 12(11), pages 1-23, November.
    13. Kostyk, Alena & Zhou, Wenkai & Hyman, Michael R., 2019. "Using surveytainment to counter declining survey data quality," Journal of Business Research, Elsevier, vol. 95(C), pages 211-219.
    14. Henderson, Isaac Levi & Tsui, Kan Wai Hong & Ngo, Thanh & Gilbey, Andrew & Avis, Mark, 2019. "Airline brand choice in a duopolistic market: The case of New Zealand," Transportation Research Part A: Policy and Practice, Elsevier, vol. 121(C), pages 147-163.
    15. Dean Spears, 2013. "Poverty and probability: aspiration and aversion to compound lotteries in El Salvador and India," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 263-284, September.
    16. Christoph Bartneck & Andreas Duenser & Elena Moltchanova & Karolina Zawieska, 2015. "Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-23, April.
    17. Emily C. Anania & Stephen Rice & Scott R. Winter & Mattie N. Milner & Nathan W. Walters & Matthew Pierce, 2018. "Why People Are Not Willing to Let Their Children Ride in Driverless School Buses: A Gender and Nationality Comparison," Social Sciences, MDPI, vol. 7(3), pages 1-17, February.
    18. Sherman, Gary D., 2020. "The moral psychology of continuation decisions: A recipe for moral disengagement," Organizational Behavior and Human Decision Processes, Elsevier, vol. 158(C), pages 36-48.
    19. Shaikh Moksadur Rahman, 2020. "Relationship between Job Satisfaction and Turnover Intention: Evidence from Bangladesh," Asian Business Review, Asian Business Consortium, vol. 10(2), pages 99-108.
    20. repec:osf:osfxxx:fqmdu_v1 is not listed on IDEAS
    21. Hind Dib‐slamani & Gilles Grolleau & Naoufel Mzoughi, 2021. "Is theft considered less severe when the victim is a foreign company?," Post-Print hal-03340844, HAL.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:cv2bn. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.