IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v46y2021i5p611-648.html
   My bibliography  Save this article

Monitoring Item Performance With CUSUM Statistics in Continuous Testing

Author

Listed:
  • Yi-Hsuan Lee
  • Charles Lewis

    (6729Educational Testing Service)

Abstract

In many educational assessments, items are reused in different administrations throughout the life of the assessments. Ideally, a reused item should perform relatively similarly over time. In reality, an item may become easier with exposure, especially when item preknowledge has occurred. This article presents a novel cumulative sum procedure for detecting item preknowledge in continuous testing where data for each reused item may be obtained from small and varying sample sizes across administrations. Its performance is evaluated with simulations and analytical work. The approach is effective in detecting item preknowledge quickly with group size at least 10 and is easy to implement with varying item parameters. In addition, it is robust to the ability estimation error introduced in the simulations.

Suggested Citation

  • Yi-Hsuan Lee & Charles Lewis, 2021. "Monitoring Item Performance With CUSUM Statistics in Continuous Testing," Journal of Educational and Behavioral Statistics, , vol. 46(5), pages 611-648, October.
    • Handle: RePEc:sae:jedbes:v:46:y:2021:i:5:p:611-648
    DOI: 10.3102/1076998621994563
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998621994563
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998621994563?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Edison M. Choe & Jinming Zhang & Hua-Hua Chang, 2018. "Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 650-673, September.
    2. Shelby Haberman & Sandip Sinharay & Kyong Chon, 2013. "Assessing Item Fit for Unidimensional Item Response Theory Models Using Residuals from Estimated Item Response Functions," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 417-440, July.
    3. Yi-Hsuan Lee & Alina Davier, 2013. "Monitoring Scale Scores over Time via Quality Control Charts, Model-Based Approaches, and Time Series Techniques," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 557-575, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chen, Yunxiao & Lee, Yi-Hsuan & Li, Xiaoou, 2022. "Item pool quality control in educational testing: change point model, compound risk, and sequential detection," LSE Research Online Documents on Economics 112498, London School of Economics and Political Science, LSE Library.
    2. Hyeon-Ah Kang, 2023. "Sequential Generalized Likelihood Ratio Tests for Online Item Monitoring," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 672-696, June.
    3. Yunxiao Chen & Yi-Hsuan Lee & Xiaoou Li, 2022. "Item Pool Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection," Journal of Educational and Behavioral Statistics, , vol. 47(3), pages 322-352, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chen, Yunxiao & Lee, Yi-Hsuan & Li, Xiaoou, 2022. "Item pool quality control in educational testing: change point model, compound risk, and sequential detection," LSE Research Online Documents on Economics 112498, London School of Economics and Political Science, LSE Library.
    2. Yunxiao Chen & Yi-Hsuan Lee & Xiaoou Li, 2022. "Item Pool Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection," Journal of Educational and Behavioral Statistics, , vol. 47(3), pages 322-352, June.
    3. Jochen Ranger & Kay Brauer, 2022. "On the Generalized S − X 2 –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization," Journal of Educational and Behavioral Statistics, , vol. 47(2), pages 202-230, April.
    4. Peter W. Rijn & Usama S. Ali, 2018. "A Generalized Speed–Accuracy Response Model for Dichotomous Items," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 109-131, March.
    5. Sandip Sinharay, 2017. "Some Remarks on Applications of Tests for Detecting A Change Point to Psychometric Problems," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1149-1161, December.
    6. Hyeon-Ah Kang, 2023. "Sequential Generalized Likelihood Ratio Tests for Online Item Monitoring," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 672-696, June.
    7. Alina Davier, 2013. "Observed-Score Equating: An Overview," Psychometrika, Springer;The Psychometric Society, vol. 78(4), pages 605-623, October.
    8. Sandip Sinharay, 2016. "Person Fit Analysis in Computerized Adaptive Testing Using Tests for a Change Point," Journal of Educational and Behavioral Statistics, , vol. 41(5), pages 521-549, October.
    9. Gerhard Tutz, 2022. "Item Response Thresholds Models: A General Class of Models for Varying Types of Items," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1238-1269, December.
    10. Yi-Hsuan Lee & Shelby Haberman, 2013. "Harmonic Regression and Scale Stability," Psychometrika, Springer;The Psychometric Society, vol. 78(4), pages 815-829, October.
    11. Onur Demirkaya & Ummugul Bezirhan & Jinming Zhang, 2023. "Detecting Item Preknowledge Using Revisits With Speed and Accuracy," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 521-542, August.
    12. Amirali Kani & Wayne S. DeSarbo & Duncan K. H. Fong, 2018. "A Factorial Hidden Markov Model for the Analysis of Temporal Change in Choice Models," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 5(3), pages 162-177, December.
    13. Hongyue Zhu & Hong Jiao & Wei Gao & Xiangbin Meng, 2023. "Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 490-520, August.
    14. Scott Monroe, 2021. "Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 374-398, June.
    15. Björn Andersson & Alina Davier, 2015. "Book Review," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 856-858, September.
    16. Yang Liu & Ji Seung Yang & Alberto Maydeu-Olivares, 2019. "Restricted Recalibration of Item Response Theory Models," Psychometrika, Springer;The Psychometric Society, vol. 84(2), pages 529-553, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:46:y:2021:i:5:p:611-648. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.