IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v50y2021i1p238-264.html
   My bibliography  Save this article

Using Instructed Response Items as Attention Checks in Web Surveys: Properties and Implementation

Author

Listed:
  • Tobias Gummer
  • Joss Roßmann
  • Henning Silber

Abstract

Identifying inattentive respondents in self-administered surveys is a challenging goal for survey researchers. Instructed response items (IRIs) provide a measure for inattentiveness in grid questions that is easy to implement. The present article adds to the sparse research on the use and implementation of attention checks by addressing three research objectives. In a first study, we provide evidence that IRIs identify respondents who show an elevated use of straightlining, speeding, item nonresponse, inconsistent answers, and implausible statements throughout a survey. Excluding inattentive respondents, however, did not alter the results of substantive analyses. Our second study suggests that respondents’ inattentiveness partially changes as the context in which they complete the survey changes. In a third study, we present experimental evidence that a mere exposure to an IRI does not negatively or positively affect response behavior within a survey. A critical discussion on using IRI attention checks concludes this article.

Suggested Citation

  • Tobias Gummer & Joss Roßmann & Henning Silber, 2021. "Using Instructed Response Items as Attention Checks in Web Surveys: Properties and Implementation," Sociological Methods & Research, , vol. 50(1), pages 238-264, February.
    • Handle: RePEc:sae:somere:v:50:y:2021:i:1:p:238-264
    DOI: 10.1177/0049124118769083
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124118769083
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124118769083?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Vera Toepoel & Corrie Vis & Marcel Das & Arthur van Soest, 2009. "Design of Web Questionnaires," Sociological Methods & Research, , vol. 37(3), pages 371-392, February.
    2. Adam J. Berinsky & Michele F. Margolis & Michael W. Sances, 2014. "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 739-753, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Olsacher, Alexandra & Bade, Celina & Ehlers, Jan & Fehring, Leonard, 2023. "How to effectively communicate health information on social media depending on the audience's personality traits: An experimental study in the context of organ donation in Germany," Social Science & Medicine, Elsevier, vol. 335(C).
    2. Panchalingam, Thadchaigeni & Howard, Gregory & Allen Klaiber, H. & Roe, Brian E., 2023. "Food choice behavior of adolescents under parent-child interaction in the context of US school lunch programs," Food Policy, Elsevier, vol. 115(C).
    3. L. Maxim Laurijssen & Barbara Wisse & Stacey Sanders & Ed Sleebos, 2024. "How to Neutralize Primary Psychopathic Leaders’ Damaging Impact: Rules, Sanctions, and Transparency," Journal of Business Ethics, Springer, vol. 189(2), pages 365-383, January.
    4. Veronika Hannus & Johannes Sauer, 2021. "Understanding Farmers’ Intention to Use a Sustainability Standard: The Role of Economic Rewards, Knowledge, and Ease of Use," Sustainability, MDPI, vol. 13(19), pages 1-21, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schiff, Kaylyn Jackson & Schiff, Daniel S. & Bueno, Natalia, 2023. "The Liar's Dividend: The Impact of Deepfakes and Fake News on Trust in Political Discourse," SocArXiv x43ph, Center for Open Science.
    2. Vecchio, Riccardo & Caso, Gerarda & Cembalo, Luigi & Borrello, Massimiliano, 2020. "Is respondents’ inattention in online surveys a major issue for research?," Economia agro-alimentare / Food Economy, Italian Society of Agri-food Economics/Società Italiana di Economia Agro-Alimentare (SIEA), vol. 22(1), March.
    3. Lude, Maximilian & Prügl, Reinhard, 2021. "Experimental studies in family business research," Journal of Family Business Strategy, Elsevier, vol. 12(1).
    4. Simon, Mark & Stanton, Steven J. & Townsend, Janell D. & Kim, John, 2019. "A multi-method study of social ties and crowdfunding success: Opening the black box to get the cash inside," Journal of Business Research, Elsevier, vol. 104(C), pages 206-214.
    5. Lindhjem, Henrik & Navrud, Ståle, 2011. "Using Internet in Stated Preference Surveys: A Review and Comparison of Survey Modes," International Review of Environmental and Resource Economics, now publishers, vol. 5(4), pages 309-351, September.
    6. de Bruijne, M.A., 2015. "Designing web surveys for the multi-device internet," Other publications TiSEM 19e4d446-a62b-4a95-8691-8, Tilburg University, School of Economics and Management.
    7. Janne Tukiainen & Sebastian Blesse & Albrecht Bohne & Leonardo M. Giuffrida & Jan Jäässkeläinen & Ari Luukinen & Antti Sieppi, 2021. "What Are the Priorities of Bureaucrats? Evidence from Conjoint Experiments with Procurement Officials," EconPol Working Paper 63, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    8. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    9. Jimin Pyo & Michael G. Maxfield, 2021. "Cognitive Effects of Inattentive Responding in an MTurk Sample," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2020-2039, July.
    10. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    11. Aljosha Henkel & Ernst Fehr & Julien Senn & Thomas Epper, 2024. "Beliefs about Inequality and the Nature of Support for Redistribution," Working Papers 2024-iRisk-02, IESEG School of Management.
    12. Le Maux, Benoît & Necker, Sarah, 2023. "Honesty nudges: Effect varies with content but not with timing," Journal of Economic Behavior & Organization, Elsevier, vol. 207(C), pages 433-456.
    13. Fehr Ernst & Epper Thomas & Senn Julien, 2020. "Social preferences and redistributive politics," ECON - Working Papers 339, Department of Economics - University of Zurich, revised Aug 2023.
    14. Lilith Burgstaller & Annabelle Doerr & Sarah Necker, 2023. "Do Household Tax Credits Increase the Demand for Legally Provided Services?," CESifo Working Paper Series 10211, CESifo.
    15. Nicole Maestas & Kathleen J. Mullen & David Powell & Till von Wachter & Jeffrey B. Wenger, 2023. "The Value of Working Conditions in the United States and the Implications for the Structure of Wages," American Economic Review, American Economic Association, vol. 113(7), pages 2007-2047, July.
    16. Burgstaller, Lilith & Doerr, Annabelle & Necker, Sarah, 2023. "Incentives for Consumers to Act as Tax Auditors: (When) Are They Effective?," VfS Annual Conference 2023 (Regensburg): Growth and the "sociale Frage" 277628, Verein für Socialpolitik / German Economic Association.
    17. Alessandro Nai, 2020. "The Trump Paradox: How Cues from a Disliked Source Foster Resistance to Persuasion," Politics and Governance, Cogitatio Press, vol. 8(1), pages 122-132.
    18. Milton Mayfield & Jacqueline Mayfield, 2021. "Sound and Safe: The Role of Leader Motivating Language and Follower Self-Leadership in Feelings of Psychological Safety," Administrative Sciences, MDPI, vol. 11(2), pages 1-30, May.
    19. Branden B. Johnson & Adam M. Finkel, 2023. "Sensitivity to scope in estimating the social benefits of prolonging lives for regulatory decisions using national stated preference tradeoffs," Environment Systems and Decisions, Springer, vol. 43(3), pages 509-528, September.
    20. Alexis Grigorieff & Christopher Roth & Diego Ubfal, 2020. "Does Information Change Attitudes Toward Immigrants?," Demography, Springer;Population Association of America (PAA), vol. 57(3), pages 1117-1143, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:50:y:2021:i:1:p:238-264. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.