IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v5y2015i2p2158244015584617.html

It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky†Tasks

Author

Listed:
  • David J. Hauser
  • Norbert Schwarz

Abstract

Instructional manipulation checks (IMCs) have become popular tools for identifying inattentive participants in online studies. IMCs function by attempting to trick inattentive participants into responding incorrectly. However, from a conversational perspective, question characteristics are part of the researcher’s contribution to the conversation, and IMCs may teach participants that there is “more than meets the eye,†prompting systematic thinking on subsequent tricky-seeming questions in an attempt to avoid being tricked. In two online studies, participants responded to a simple task either before or after completing an IMC. As expected, answering an IMC prior to the task improved performance on items that benefit from increased systematic thinking—namely, the Cognitive Reflection Test (Study 1), and a probabilistic reasoning task (Study 2). We conclude that IMCs change attention rather than merely measure attention and discuss implications for their use in online studies.

Suggested Citation

  • David J. Hauser & Norbert Schwarz, 2015. "It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky†Tasks," SAGE Open, , vol. 5(2), pages 21582440155, April.
    • Handle: RePEc:sae:sagope:v:5:y:2015:i:2:p:2158244015584617
    DOI: 10.1177/2158244015584617
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244015584617
    Download Restriction: no
    File URL: https://libkey.io/10.1177/2158244015584617?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Adam J. Berinsky & Michele F. Margolis & Michael W. Sances, 2014. "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 739-753, July.
    2. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    3. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Olbrich, Lukas & Sakshaug, Joseph W. & Lewandowski, Eric, 2024. "Evaluating methods to prevent and detect inattentive respondents in web surveys," SocArXiv py9gz, Center for Open Science.
    2. Eugene Y. Chan & Jack Lin, 2022. "Political ideology and psychological reactance: how serious should climate change be?," Climatic Change, Springer, vol. 172(1), pages 1-22, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    2. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    3. Brañas-Garza, Pablo & Kujal, Praveen & Lenkei, Balint, 2019. "Cognitive reflection test: Whom, how, when," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 82(C).
    4. Irene Scopelliti & Carey K. Morewedge & Erin McCormick & H. Lauren Min & Sophie Lebrecht & Karim S. Kassam, 2015. "Bias Blind Spot: Structure, Measurement, and Consequences," Management Science, INFORMS, vol. 61(10), pages 2468-2486, October.
    5. Ma-Kellams, Christine & Lerner, Jennifer, 2016. "Trust your gut or think carefully? Examining whether an intuitive, versus a systematic, mode of thought produces greater empathic accuracy," Scholarly Articles 37093806, Harvard Kennedy School of Government.
    6. Spears, Dean, 2019. "The Asymmetry of Population Ethics: Experimental Social Choice and Dual-Process Moral Reasoning," IZA Discussion Papers 12537, IZA Network @ LISER.
    7. Ronayne, David & Sgroi, Daniel & Tuckwell, Anthony, 2021. "Evaluating the sunk cost effect," Journal of Economic Behavior & Organization, Elsevier, vol. 186(C), pages 318-327.
    8. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    9. Jeremy Cone & David G Rand, 2014. "Time Pressure Increases Cooperation in Competitively Framed Social Dilemmas," PLOS ONE, Public Library of Science, vol. 9(12), pages 1-13, December.
    10. David G Rand & Gordon Kraft-Todd & June Gruber, 2015. "The Collective Benefits of Feeling Good and Letting Go: Positive Emotion and (dis)Inhibition Interact to Predict Cooperative Behavior," PLOS ONE, Public Library of Science, vol. 10(1), pages 1-12, January.
    11. Yariv, Leeat & Snowberg, Erik, 2018. "Testing the Waters: Behavior across Participant Pools," CEPR Discussion Papers 13015, C.E.P.R. Discussion Papers.
    12. Ma-Kellams, Christine & Lerner, Jennifer S., 2016. "Trust Your Gut or Think Carefully? Examining Whether an Intuitive versus a Systematic Mode of Thought Produces Greater Empathic Accuracy," Working Paper Series 16-017, Harvard University, John F. Kennedy School of Government.
    13. Acar-Burkay, Sinem & Cristian, Daniela-Carmen, 2022. "Cognitive underpinnings of COVID-19 vaccine hesitancy," Social Science & Medicine, Elsevier, vol. 301(C).
    14. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.
    15. Irene Scopelliti & H. Lauren Min & Erin McCormick & Karim S. Kassam & Carey K. Morewedge, 2018. "Individual Differences in Correspondence Bias: Measurement, Consequences, and Correction of Biased Interpersonal Attributions," Management Science, INFORMS, vol. 64(4), pages 1879-1910, April.
    16. repec:plo:pone00:0226394 is not listed on IDEAS
    17. Dean Spears, 2013. "Poverty and probability: aspiration and aversion to compound lotteries in El Salvador and India," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 263-284, September.
    18. Prissé, Benjamin & Jorrat, Diego, 2021. "Lack of Control: An experiment," MPRA Paper 109918, University Library of Munich, Germany.
    19. Del Ponte, Alessandro & Li, Lianjun & Ang, Lina & Lim, Noah & Seow, Wei Jie, 2024. "Evaluating SoJump.com as a tool for online behavioral research in China," Journal of Behavioral and Experimental Finance, Elsevier, vol. 41(C).
    20. Erik Snowberg & Leeat Yariv, 2018. "Testing the Waters: Behavior across Participant Pools," CESifo Working Paper Series 7136, CESifo.
    21. Ronayne, David & Sgroi, Daniel & Tuckwell, Anthony, 2020. "Evaluating the Sunk Cost Effect," The Warwick Economics Research Paper Series (TWERPS) 1269, University of Warwick, Department of Economics.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:5:y:2015:i:2:p:2158244015584617. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.