IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v59y2025i1d10.1007_s11135-024-01973-z.html
   My bibliography  Save this article

Online surveys: lessons learned in detecting and protecting against insincerity and bots

Author

Listed:
  • Amber D. Thompson

    (University of Utah)

  • Rebecca L. Utz

    (University of Utah)

Abstract

Online survey panels (e.g., KnowledgePanel) and large-scale survey programs (e.g., World Values Survey) commonly used by social scientists may not support research on rare populations. For researchers interested in accessing specialized populations, openly accessible invitation links conducted in web mode may be necessary. However, surveys recruiting through online channels and administered online are vulnerable to infiltration by bots and insincere responses, and developing procedures and strategies for identifying and filtering out fraudulent responses is a pressing concern for this typeof survey research. This article describes our experience with an infiltrated sample, and provides a multifaceted data cleaning process that can be used to identify insincereand bot responses. Insincere and bot responses are becoming more sophisticated and less likely to be automatically detected by online survey vendors. Employing a multifaceted systematic methodology during the data cleaning stage, is recommended to ensure the validity and integrity of open-access survey data collected online.

Suggested Citation

  • Amber D. Thompson & Rebecca L. Utz, 2025. "Online surveys: lessons learned in detecting and protecting against insincerity and bots," Quality & Quantity: International Journal of Methodology, Springer, vol. 59(1), pages 23-39, February.
    • Handle: RePEc:spr:qualqt:v:59:y:2025:i:1:d:10.1007_s11135-024-01973-z
    DOI: 10.1007/s11135-024-01973-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11135-024-01973-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11135-024-01973-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Daniel Schneider & Kristen Harknett, 2022. "What’s to Like? Facebook as a Tool for Survey Data Collection," Sociological Methods & Research, , vol. 51(1), pages 108-140, February.
    2. Jennifer P. Agans & Serena A. Schade & Steven R. Hanna & Shou-Chun Chiang & Kimia Shirzad & Sunhye Bai, 2024. "The inaccuracy of data from online surveys: A cautionary analysis," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(3), pages 2065-2086, June.
    3. Boas, Taylor C. & Christenson, Dino P. & Glick, David M., 2020. "Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics," Political Science Research and Methods, Cambridge University Press, vol. 8(2), pages 232-250, April.
    4. Marybec Griffin & Richard J. Martino & Caleb LoSchiavo & Camilla Comer-Carruthers & Kristen D. Krause & Christopher B. Stults & Perry N. Halkitis, 2022. "Ensuring survey research data integrity in the era of internet bots," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(4), pages 2841-2852, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Barton, Jared & Pan, Xiaofei, 2022. "Movin’ on up? A survey experiment on mobility enhancing policies," European Journal of Political Economy, Elsevier, vol. 74(C).
    2. Robert Kubinec & Haillie Na‐Kyung Lee & Andrey Tomashevskiy, 2021. "Politically connected companies are less likely to shutdown due to COVID‐19 restrictions," Social Science Quarterly, Southwestern Social Science Association, vol. 102(5), pages 2155-2169, September.
    3. Narae Kim & Jeong-Nam Kim, 2024. "A COVID-19 Paradox of Communication, Ignorance, and Vaccination Intention," SAGE Open, , vol. 14(3), pages 21582440241, September.
    4. Ji Yun Lee & Fangjiao Ma & Yue Li, 2022. "Understanding homeowner proactive actions for managing wildfire risks," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 114(2), pages 1525-1547, November.
    5. Michael Kumove, 2022. "Does Language Foster Reconciliation? Evidence From the Former Yugoslavia," Journal of Conflict Resolution, Peace Science Society (International), vol. 66(4-5), pages 783-808, May.
    6. Lisa M. Fucito & Krysten W. Bold & Sydney Cannon & Alison Serrantino & Rebecca Marrero & Stephanie S. O’Malley, 2022. "Cigarette Smoking in Response to COVID-19: Examining Co-Morbid Medical Conditions and Risk Perceptions," IJERPH, MDPI, vol. 19(14), pages 1-12, July.
    7. Sean F. Ellis & Olesya M. Savchenko & Kent D. Messer, 2022. "Mitigating stigma associated with recycled water," American Journal of Agricultural Economics, John Wiley & Sons, vol. 104(3), pages 1077-1099, May.
    8. Oguzhan Essiz & Aysu Senyuz, 2024. "Predicting the value‐based determinants of sustainable luxury consumption: A multi‐analytical approach and pathway to sustainable development in the luxury industry," Business Strategy and the Environment, Wiley Blackwell, vol. 33(3), pages 1721-1758, March.
    9. Hannah Casey & Deirdre Desmond & Laura Coffey, 2023. "Professional and Family Carers’ Perspectives on the COVID-19 Pandemic and Its Impact on Supported Decision-Making with Adults with Intellectual Disabilities: A Qualitative Online Survey," Disabilities, MDPI, vol. 3(2), pages 1-11, April.
    10. Matthew H. Goldberg & Abel Gustafson & Matthew T. Ballew & Seth A. Rosenthal & Matthew J. Cutler & Anthony Leiserowitz, 2020. "Predictors of global warming risk perceptions among Latino and non-Latino White Americans," Climatic Change, Springer, vol. 162(3), pages 1555-1574, October.
    11. Jie Zhao & Ji Yun Lee & Guirong Yan & Fangjiao Ma, 2024. "Quantifying the role of insurance in tornado-impacted community recovery: a survey and simulation-based approach," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 120(8), pages 7435-7459, June.
    12. Heidi Reed, 2022. "When the Right Thing to Do Is Also the Wrong Thing: Moral Sensemaking of Responsible Business Behavior During the COVID-19 Crisis," Post-Print hal-04531082, HAL.
    13. Natasha V. Pilkauskas & Katherine Michelmore & Nicole Kovski & H. Luke Shaefer, 2024. "The expanded Child Tax Credit and economic wellbeing of low-income families," Journal of Population Economics, Springer;European Society for Population Economics, vol. 37(4), pages 1-35, December.
    14. Huichuan Xia, 2023. "What scholars and IRBs talk when they talk about the Belmont principles in crowd work‐based research," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(1), pages 67-80, January.
    15. Roselyn J. Lee-Won & Inyoung Jang & Hyun-Suk Kim & Sung-Gwan Park, 2022. "The Relationship between Future Anxiety Due to COVID-19 and Vigilance: The Role of Message Fatigue and Autonomy Satisfaction," IJERPH, MDPI, vol. 19(3), pages 1-16, January.
    16. Xuesong You & Carolyn Kousky, 2024. "Improving household and community disaster recovery: Evidence on the role of insurance," Journal of Risk & Insurance, The American Risk and Insurance Association, vol. 91(2), pages 299-338, June.
    17. Sarpong, Dennis, 2024. "National Identification and Public Cooperation with the Police in Ghana," Journal of Criminal Justice, Elsevier, vol. 94(C).
    18. Pankaj Tiwari, 2023. "Influence of Millennials’ eco-literacy and biospheric values on green purchases: the mediating effect of attitude," Public Organization Review, Springer, vol. 23(3), pages 1195-1212, September.
    19. S. Brent Jackson & Kathryn T. Stevenson & Lincoln R. Larson & M. Nils Peterson & Erin Seekamp, 2021. "Outdoor Activity Participation Improves Adolescents’ Mental Health and Well-Being during the COVID-19 Pandemic," IJERPH, MDPI, vol. 18(5), pages 1-18, March.
    20. Park, Keunhyun & Chamberlain, Brent & Song, Ziqi & Nasr Esfahani, Hossein & Sheen, Jeff & Larsen, Teresa & Long Novack, Valerie & Licon, Carlos & Christensen, Keith, 2022. "A double jeopardy: COVID-19 impacts on the travel behavior and community living of people with disabilities," Transportation Research Part A: Policy and Practice, Elsevier, vol. 156(C), pages 24-35.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:59:y:2025:i:1:d:10.1007_s11135-024-01973-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.