IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v88y2023i2d10.1007_s11336-022-09871-9.html
   My bibliography  Save this article

Sequential Generalized Likelihood Ratio Tests for Online Item Monitoring

Author

Listed:
  • Hyeon-Ah Kang

    (University of Texas at Austin)

Abstract

The study presents statistical procedures that monitor functioning of items over time. We propose generalized likelihood ratio tests that surveil multiple item parameters and implement with various sampling techniques to perform continuous or intermittent monitoring. The procedures examine stability of item parameters across time and inform compromise as soon as they identify significant parameter shift. The performance of the monitoring procedures was validated using simulated and real-assessment data. The empirical evaluation suggests that the proposed procedures perform adequately well in identifying the parameter drift. They showed satisfactory detection power and gave timely signals while regulating error rates reasonably low. The procedures also showed superior performance when compared with the existent methods. The empirical findings suggest that multivariate parametric monitoring can provide an efficient and powerful control tool for maintaining the quality of items. The procedures allow joint monitoring of multiple item parameters and achieve sufficient power using powerful likelihood-ratio tests. Based on the findings from the empirical experimentation, we suggest some practical strategies for performing online item monitoring.

Suggested Citation

  • Hyeon-Ah Kang, 2023. "Sequential Generalized Likelihood Ratio Tests for Online Item Monitoring," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 672-696, June.
    • Handle: RePEc:spr:psycho:v:88:y:2023:i:2:d:10.1007_s11336-022-09871-9
    DOI: 10.1007/s11336-022-09871-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-022-09871-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-022-09871-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wim van der Linden, 2007. "A Hierarchical Framework for Modeling Speed and Accuracy on Test Items," Psychometrika, Springer;The Psychometric Society, vol. 72(3), pages 287-308, September.
    2. Xi Wang & Yang Liu, 2020. "Detecting Compromised Items Using Information From Secure Items," Journal of Educational and Behavioral Statistics, , vol. 45(6), pages 667-689, December.
    3. Yi-Hsuan Lee & Charles Lewis, 2021. "Monitoring Item Performance With CUSUM Statistics in Continuous Testing," Journal of Educational and Behavioral Statistics, , vol. 46(5), pages 611-648, October.
    4. Edison M. Choe & Jinming Zhang & Hua-Hua Chang, 2018. "Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 650-673, September.
    5. Zhang, Jiujun & Li, Zhonghua & Wang, Zhaojun, 2010. "A multivariate control chart for simultaneously monitoring process mean and variability," Computational Statistics & Data Analysis, Elsevier, vol. 54(10), pages 2244-2252, October.
    6. Hyeon-Ah Kang & Yi Zheng & Hua-Hua Chang, 2020. "Online Calibration of a Joint Model of Item Responses and Response Times in Computerized Adaptive Testing," Journal of Educational and Behavioral Statistics, , vol. 45(2), pages 175-208, April.
    7. Zhan Shu & Robert Henson & Richard Luecht, 2013. "Using Deterministic, Gated Item Response Theory Model to Detect Test Cheating due to Item Compromise," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 481-497, July.
    8. Wim Linden & Fanmin Guo, 2008. "Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 73(3), pages 365-384, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chun Wang & Gongjun Xu & Zhuoran Shang, 2018. "A Two-Stage Approach to Differentiating Normal and Aberrant Behavior in Computer Based Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 223-254, March.
    2. Onur Demirkaya & Ummugul Bezirhan & Jinming Zhang, 2023. "Detecting Item Preknowledge Using Revisits With Speed and Accuracy," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 521-542, August.
    3. Hongyue Zhu & Hong Jiao & Wei Gao & Xiangbin Meng, 2023. "Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 490-520, August.
    4. Chen, Yunxiao & Lu, Yan & Moustaki, Irini, 2022. "Detection of two-way outliers in multivariate data and application to cheating detection in educational tests," LSE Research Online Documents on Economics 112499, London School of Economics and Political Science, LSE Library.
    5. Edison M. Choe & Jinming Zhang & Hua-Hua Chang, 2018. "Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 650-673, September.
    6. Steffi Pohl & Esther Ulitzsch & Matthias Davier, 2019. "Using Response Times to Model Not-Reached Items due to Time Limits," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 892-920, September.
    7. Maria Bolsinova & Paul Boeck & Jesper Tijmstra, 2017. "Modelling Conditional Dependence Between Response Time and Accuracy," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1126-1148, December.
    8. Dylan Molenaar & Paul Boeck, 2018. "Response Mixture Modeling: Accounting for Heterogeneity in Item Characteristics across Response Times," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 279-297, June.
    9. Sandip Sinharay & Peter W. van Rijn, 2020. "Assessing Fit of the Lognormal Model for Response Times," Journal of Educational and Behavioral Statistics, , vol. 45(5), pages 534-568, October.
    10. Renske E. Kuijpers & Ingmar Visser & Dylan Molenaar, 2021. "Testing the Within-State Distribution in Mixture Models for Responses and Response Times," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 348-373, June.
    11. Chen, Yunxiao & Lee, Yi-Hsuan & Li, Xiaoou, 2022. "Item pool quality control in educational testing: change point model, compound risk, and sequential detection," LSE Research Online Documents on Economics 112498, London School of Economics and Political Science, LSE Library.
    12. Wim J. van der Linden, 2009. "A Bivariate Lognormal Response-Time Model for the Detection of Collusion Between Test Takers," Journal of Educational and Behavioral Statistics, , vol. 34(3), pages 378-394, September.
    13. Yue Liu & Hongyun Liu, 2021. "Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process," Journal of Educational and Behavioral Statistics, , vol. 46(6), pages 717-752, December.
    14. Yunxiao Chen & Yi-Hsuan Lee & Xiaoou Li, 2022. "Item Pool Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection," Journal of Educational and Behavioral Statistics, , vol. 47(3), pages 322-352, June.
    15. Hyeon-Ah Kang & Yi Zheng & Hua-Hua Chang, 2020. "Online Calibration of a Joint Model of Item Responses and Response Times in Computerized Adaptive Testing," Journal of Educational and Behavioral Statistics, , vol. 45(2), pages 175-208, April.
    16. Matthias von Davier & Lale Khorramdel & Qiwei He & Hyo Jeong Shin & Haiwen Chen, 2019. "Developments in Psychometric Population Models for Technology-Based Large-Scale Assessments: An Overview of Challenges and Opportunities," Journal of Educational and Behavioral Statistics, , vol. 44(6), pages 671-705, December.
    17. Steven Andrew Culpepper & James Joseph Balamuta, 2017. "A Hierarchical Model for Accuracy and Choice on Standardized Tests," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 820-845, September.
    18. Inhan Kang & Paul Boeck & Roger Ratcliff, 2022. "Modeling Conditional Dependence of Response Accuracy and Response Time with the Diffusion Item Response Theory Model," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 725-748, June.
    19. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    20. Jochen Ranger, 2013. "A Note on the Hierarchical Model for Responses and Response Times in Tests of van der Linden (2007)," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 538-544, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:88:y:2023:i:2:d:10.1007_s11336-022-09871-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.