IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v44y2019i3p309-341.html
   My bibliography  Save this article

Detection and Treatment of Careless Responses to Improve Item Parameter Estimation

Author

Listed:
  • Jeffrey M. Patton

    (Financial Industry Regulatory Authority (FINRA))

  • Ying Cheng
  • Maxwell Hong

    (University of Notre Dame)

  • Qi Diao

    (Educational Testing Service)

Abstract

In psychological and survey research, the prevalence and serious consequences of careless responses from unmotivated participants are well known. In this study, we propose to iteratively detect careless responders and cleanse the data by removing their responses. The careless responders are detected using person-fit statistics. In two simulation studies, the iterative procedure leads to nearly perfect power in detecting extremely careless responders and much higher power than the noniterative procedure in detecting moderately careless responders. Meanwhile, the false-positive error rate is close to the nominal level. In addition, item parameter estimation is much improved by iteratively cleansing the calibration sample. The bias in item discrimination and location parameter estimates is substantially reduced. The standard error estimates, which are spuriously small in the presence of careless responses, are corrected by the iterative cleansing procedure. An empirical example is also presented to illustrate the proposed procedure. These results suggest that the proposed procedure is a promising way to improve item parameter estimation for tests of 20 items or longer when data are contaminated by careless responses.

Suggested Citation

  • Jeffrey M. Patton & Ying Cheng & Maxwell Hong & Qi Diao, 2019. "Detection and Treatment of Careless Responses to Improve Item Parameter Estimation," Journal of Educational and Behavioral Statistics, , vol. 44(3), pages 309-341, June.
    • Handle: RePEc:sae:jedbes:v:44:y:2019:i:3:p:309-341
    DOI: 10.3102/1076998618825116
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998618825116
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998618825116?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(4), pages 691-705, August.
    2. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(5), pages 879-883, October.
    3. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    4. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(6), pages 1195-1198, December.
    5. Thomas Warm, 1989. "Weighted likelihood estimation of ability in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 54(3), pages 427-450, September.
    6. Tom Snijders, 2001. "Asymptotic null distribution of person fit statistics with estimated person parameter," Psychometrika, Springer;The Psychometric Society, vol. 66(3), pages 331-342, September.
    7. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(1), pages 225-228, February.
    8. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(2), pages 411-413, April.
    9. Can Shao & Jun Li & Ying Cheng, 2016. "Detection of Test Speededness Using Change-Point Analysis," Psychometrika, Springer;The Psychometric Society, vol. 81(4), pages 1118-1141, December.
    10. C. Glas & Anna Dagohoy, 2007. "A Person Fit Test For Irt Models For Polytomous Items," Psychometrika, Springer;The Psychometric Society, vol. 72(2), pages 159-180, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yue Liu & Hongyun Liu, 2021. "Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process," Journal of Educational and Behavioral Statistics, , vol. 46(6), pages 717-752, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yakut, Oguz, 2021. "Implementation of hydraulically driven barrel shooting control by utilizing artificial neural networks," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 190(C), pages 1206-1223.
    2. X. Qin & G. Huang, 2009. "An Inexact Chance-constrained Quadratic Programming Model for Stream Water Quality Management," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 23(4), pages 661-695, March.
    3. Md. Yousuf Gazi & Khandakar Tahmida Tafhim, 2019. "Investigation of Heavy-mineral Deposits Using Multispectral Satellite Imagery in the Eastern Coastal Margin of Bangladesh," Earth Sciences Malaysia (ESMY), Zibeline International Publishing, vol. 3(2), pages 16-22, October.
    4. Billionnet, Alain, 2011. "Solving the probabilistic reserve selection problem," Ecological Modelling, Elsevier, vol. 222(3), pages 546-554.
    5. Minghe Sun, 2005. "Warm-Start Routines for Solving Augmented Weighted Tchebycheff Network Programs in Multiple-Objective Network Programming," INFORMS Journal on Computing, INFORMS, vol. 17(4), pages 422-437, November.
    6. François Clautiaux & Cláudio Alves & José Valério de Carvalho & Jürgen Rietz, 2011. "New Stabilization Procedures for the Cutting Stock Problem," INFORMS Journal on Computing, INFORMS, vol. 23(4), pages 530-545, November.
    7. Eichengreen, Barry & Kletzer, Kenneth & Mody, Ashoka, 2003. "Crisis Resolution: Next Steps," Santa Cruz Center for International Economics, Working Paper Series qt4cj974r4, Center for International Economics, UC Santa Cruz.
    8. Tansel, Aysit & Karao?lan, Deniz, 2016. "The Causal Effect of Education on Health Behaviors: Evidence from Turkey," IZA Discussion Papers 10020, Institute of Labor Economics (IZA).
    9. Di Feng & Bettina Klaus, 2022. "Preference revelation games and strict cores of multiple‐type housing market problems," International Journal of Economic Theory, The International Society for Economic Theory, vol. 18(1), pages 61-76, March.
    10. Anna Scherbina, 2021. "Assessing the Optimality of a COVID Lockdown in the United States," Economics of Disasters and Climate Change, Springer, vol. 5(2), pages 177-201, July.
    11. John McKay, 2005. "How Significant and Effective are North Korea's "Market Reforms"?," Global Economic Review, Taylor & Francis Journals, vol. 34(1), pages 83-97.
    12. Timothy K.M. Beatty & Erling Røed Larsen & Dag Einar Sommervoll, 2005. "Measuring the Price of Housing Consumption for Owners in the CPI," Discussion Papers 427, Statistics Norway, Research Department.
    13. Marco Bianchi & Carlos Tapia & Ikerne del Valle, 2020. "Monitoring domestic material consumption at lower territorial levels: A novel data downscaling method," Journal of Industrial Ecology, Yale University, vol. 24(5), pages 1074-1087, October.
    14. Sonmez, Tayfun & Utku Unver, M., 2005. "House allocation with existing tenants: an equivalence," Games and Economic Behavior, Elsevier, vol. 52(1), pages 153-185, July.
    15. Juarez, Ruben, 2013. "Group strategyproof cost sharing: The role of indifferences," Games and Economic Behavior, Elsevier, vol. 82(C), pages 218-239.
    16. Bustillo, Inés & Velloso, Helvia & Vézina, François, 2006. "The Canadian retirement income system," Documentos de Proyectos 3682, Naciones Unidas Comisión Económica para América Latina y el Caribe (CEPAL).
    17. Melega, Gislaine Mara & de Araujo, Silvio Alexandre & Jans, Raf, 2018. "Classification and literature review of integrated lot-sizing and cutting stock problems," European Journal of Operational Research, Elsevier, vol. 271(1), pages 1-19.
    18. Roth, Alvin E. & Sonmez, Tayfun & Utku Unver, M., 2005. "Pairwise kidney exchange," Journal of Economic Theory, Elsevier, vol. 125(2), pages 151-188, December.
    19. Martino Bardi & Peter Caines & Italo Capuzzo Dolcetta, 2013. "Preface: DGAA Special Issue on Mean Field Games," Dynamic Games and Applications, Springer, vol. 3(4), pages 443-445, December.
    20. repec:dau:papers:123456789/5389 is not listed on IDEAS
    21. Robert Hahn & Paul Tetlock, 2006. "A New Approach for Regulating Information Markets," Journal of Regulatory Economics, Springer, vol. 29(3), pages 265-281, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:44:y:2019:i:3:p:309-341. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.