IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v44y2019i3p309-341.html
   My bibliography  Save this article

Detection and Treatment of Careless Responses to Improve Item Parameter Estimation

Author

Listed:
  • Jeffrey M. Patton

    (Financial Industry Regulatory Authority (FINRA))

  • Ying Cheng
  • Maxwell Hong

    (University of Notre Dame)

  • Qi Diao

    (Educational Testing Service)

Abstract

In psychological and survey research, the prevalence and serious consequences of careless responses from unmotivated participants are well known. In this study, we propose to iteratively detect careless responders and cleanse the data by removing their responses. The careless responders are detected using person-fit statistics. In two simulation studies, the iterative procedure leads to nearly perfect power in detecting extremely careless responders and much higher power than the noniterative procedure in detecting moderately careless responders. Meanwhile, the false-positive error rate is close to the nominal level. In addition, item parameter estimation is much improved by iteratively cleansing the calibration sample. The bias in item discrimination and location parameter estimates is substantially reduced. The standard error estimates, which are spuriously small in the presence of careless responses, are corrected by the iterative cleansing procedure. An empirical example is also presented to illustrate the proposed procedure. These results suggest that the proposed procedure is a promising way to improve item parameter estimation for tests of 20 items or longer when data are contaminated by careless responses.

Suggested Citation

  • Jeffrey M. Patton & Ying Cheng & Maxwell Hong & Qi Diao, 2019. "Detection and Treatment of Careless Responses to Improve Item Parameter Estimation," Journal of Educational and Behavioral Statistics, , vol. 44(3), pages 309-341, June.
    • Handle: RePEc:sae:jedbes:v:44:y:2019:i:3:p:309-341
    DOI: 10.3102/1076998618825116
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998618825116
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998618825116?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Tom Snijders, 2001. "Asymptotic null distribution of person fit statistics with estimated person parameter," Psychometrika, Springer;The Psychometric Society, vol. 66(3), pages 331-342, September.
    2. C. Glas & Anna Dagohoy, 2007. "A Person Fit Test For Irt Models For Polytomous Items," Psychometrika, Springer;The Psychometric Society, vol. 72(2), pages 159-180, June.
    3. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    4. Can Shao & Jun Li & Ying Cheng, 2016. "Detection of Test Speededness Using Change-Point Analysis," Psychometrika, Springer;The Psychometric Society, vol. 81(4), pages 1118-1141, December.
    5. Thomas Warm, 1989. "Weighted likelihood estimation of ability in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 54(3), pages 427-450, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yue Liu & Hongyun Liu, 2021. "Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process," Journal of Educational and Behavioral Statistics, , vol. 46(6), pages 717-752, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maxwell Hong & Lizhen Lin & Ying Cheng, 2021. "Asymptotically Corrected Person Fit Statistics for Multidimensional Constructs with Simple Structure and Mixed Item Types," Psychometrika, Springer;The Psychometric Society, vol. 86(2), pages 464-488, June.
    2. David Magis & Gilles Raîche & Sébastien Béland, 2012. "A Didactic Presentation of Snijders’s lz* Index of Person Fit With Emphasis on Response Model Selection and Ability Estimation," Journal of Educational and Behavioral Statistics, , vol. 37(1), pages 57-81, February.
    3. Sandip Sinharay, 2016. "Person Fit Analysis in Computerized Adaptive Testing Using Tests for a Change Point," Journal of Educational and Behavioral Statistics, , vol. 41(5), pages 521-549, October.
    4. Sandip Sinharay, 2016. "Asymptotically Correct Standardization of Person-Fit Statistics Beyond Dichotomous Items," Psychometrika, Springer;The Psychometric Society, vol. 81(4), pages 992-1013, December.
    5. Sandip Sinharay, 2017. "Detection of Item Preknowledge Using Likelihood Ratio Test and Score Test," Journal of Educational and Behavioral Statistics, , vol. 42(1), pages 46-68, February.
    6. Xiang Liu & James Yang & Hui Soo Chae & Gary Natriello, 2020. "Power Divergence Family of Statistics for Person Parameters in IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 502-525, June.
    7. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    8. Klaas Sijtsma & Jules L. Ellis & Denny Borsboom, 2024. "Recognize the Value of the Sum Score, Psychometrics’ Greatest Accomplishment," Psychometrika, Springer;The Psychometric Society, vol. 89(1), pages 84-117, March.
    9. Kylie Gorney & Sandip Sinharay & Carol Eckerly, 2024. "Efficient Corrections for Standardized Person-Fit Statistics," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 569-591, June.
    10. Elina Tsigeman & Sebastian Silas & Klaus Frieler & Maxim Likhanov & Rebecca Gelding & Yulia Kovas & Daniel Müllensiefen, 2022. "The Jack and Jill Adaptive Working Memory Task: Construction, Calibration and Validation," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-29, January.
    11. Alexander Robitzsch, 2021. "A Comprehensive Simulation Study of Estimation Methods for the Rasch Model," Stats, MDPI, vol. 4(4), pages 1-23, October.
    12. Sandip Sinharay, 2015. "The Asymptotic Distribution of Ability Estimates," Journal of Educational and Behavioral Statistics, , vol. 40(5), pages 511-528, October.
    13. Kevin Carl P. Santos & Jimmy Torre & Matthias Davier, 2020. "Adjusting Person Fit Index for Skewness in Cognitive Diagnosis Modeling," Journal of Classification, Springer;The Classification Society, vol. 37(2), pages 399-420, July.
    14. Sandip Sinharay & Jens Ledet Jensen, 2019. "Higher-Order Asymptotics and Its Application to Testing the Equality of the Examinee Ability Over Two Sets of Items," Psychometrika, Springer;The Psychometric Society, vol. 84(2), pages 484-510, June.
    15. Carmen Köhler & Alexander Robitzsch & Johannes Hartig, 2020. "A Bias-Corrected RMSD Item Fit Statistic: An Evaluation and Comparison to Alternatives," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 251-273, June.
    16. Sandip Sinharay & Matthew S. Johnson, 2021. "The Use of the Posterior Probability in Score Differencing," Journal of Educational and Behavioral Statistics, , vol. 46(4), pages 403-429, August.
    17. Zhongtian Lin & Tao Jiang & Frank Rijmen & Paul Wamelen, 2024. "Asymptotically Correct Person Fit z-Statistics For the Rasch Testlet Model," Psychometrika, Springer;The Psychometric Society, vol. 89(4), pages 1230-1260, December.
    18. Felix Zimmer & Clemens Draxler & Rudolf Debelak, 2023. "Power Analysis for the Wald, LR, Score, and Gradient Tests in a Marginal Maximum Likelihood Framework: Applications in IRT," Psychometrika, Springer;The Psychometric Society, vol. 88(4), pages 1249-1298, December.
    19. Peter W. Rijn & Usama S. Ali & Hyo Jeong Shin & Sean-Hwane Joo, 2024. "Adjusted Residuals for Evaluating Conditional Independence in IRT Models for Multistage Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 89(1), pages 317-346, March.
    20. J. R. Lockwood & D. McCaffrey, 2020. "Using hidden information and performance level boundaries to study student–teacher assignments: implications for estimating teacher causal effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(4), pages 1333-1362, October.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:44:y:2019:i:3:p:309-341. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.