IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v48y2023i6p842-865.html
   My bibliography  Save this article

Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Author

Listed:
  • Yan Li

    (Shaanxi Normal University)

  • Chao Huang

    (Benben Educational Technology (Beijing) Company)

  • Jia Liu

    (Tsinghua University)

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a cutting-edge technology in educational measurement that targets at providing feedback on examinees’ strengths and weaknesses while increasing test accuracy and efficiency. To date, most CD-CAT studies have made methodological progress under simulated conditions, but little has applied CD-CAT to real educational assessment. The present study developed a Chinese reading comprehension item bank tapping into six validated reading attributes, with 195 items calibrated using data of 28,485 second to sixth graders and the item-level cognitive diagnostic models (CDMs). The measurement precision and efficiency of the reading CD-CAT system were compared and optimized in terms of crucial CD-CAT settings, including the CDMs for calibration, item selection methods, and termination rules. The study identified seven dominant reading attribute mastery profiles that stably exist across grades. These major clusters of readers and their variety with grade indicated some sort of reading developmental mechanisms that advance and deepen step by step at the primary school level. Results also suggested that compared to traditional linear tests, CD-CAT significantly improved the classification accuracy without imposing much testing burden. These findings may elucidate the multifaceted nature and possible learning paths of reading and raise the question of whether CD-CAT is applicable to other educational domains where there is a need to provide formative and fine-grained feedback but where there is a limited amount of test time.

Suggested Citation

  • Yan Li & Chao Huang & Jia Liu, 2023. "Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?," Journal of Educational and Behavioral Statistics, , vol. 48(6), pages 842-865, December.
    • Handle: RePEc:sae:jedbes:v:48:y:2023:i:6:p:842-865
    DOI: 10.3102/10769986231160668
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986231160668
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986231160668?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hua-Hua Chang, 2015. "Psychometrics Behind Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 1-20, March.
    2. Hong-Yun Liu & Xiao-Feng You & Wen-Yi Wang & Shu-Liang Ding & Hua-Hua Chang, 2013. "The Development of Computerized Adaptive Testing with Cognitive Diagnosis for an English Achievement Test in China," Journal of Classification, Springer;The Classification Society, vol. 30(2), pages 152-172, July.
    3. Chia-Yi Chiu & Yuan-Pei Chang, 2021. "Advances in CD-CAT: The General Nonparametric Item Selection Method," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 1039-1057, December.
    4. Chun Wang & Jing Lu, 2021. "Learning Attribute Hierarchies From Data: Two Exploratory Approaches," Journal of Educational and Behavioral Statistics, , vol. 46(1), pages 58-84, February.
    5. Ying Cheng, 2009. "When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT," Psychometrika, Springer;The Psychometric Society, vol. 74(4), pages 619-632, December.
    6. Peida Zhan & Hong Jiao & Dandan Liao & Feiming Li, 2019. "A Longitudinal Higher-Order Diagnostic Classification Model," Journal of Educational and Behavioral Statistics, , vol. 44(3), pages 251-281, June.
    7. Wenyi Wang & Lihong Song & Teng Wang & Peng Gao & Jian Xiong, 2020. "A Note on the Relationship of the Shannon Entropy Procedure and the Jensen–Shannon Divergence in Cognitive Diagnostic Computerized Adaptive Testing," SAGE Open, , vol. 10(1), pages 21582440198, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Qingrong Tan & Yan Cai & Fen Luo & Dongbo Tu, 2023. "Development of a High-Accuracy and Effective Online Calibration Method in CD-CAT Based on Gini Index," Journal of Educational and Behavioral Statistics, , vol. 48(1), pages 103-141, February.
    2. Wenyi Wang & Lihong Song & Teng Wang & Peng Gao & Jian Xiong, 2020. "A Note on the Relationship of the Shannon Entropy Procedure and the Jensen–Shannon Divergence in Cognitive Diagnostic Computerized Adaptive Testing," SAGE Open, , vol. 10(1), pages 21582440198, January.
    3. Hung-Yu Huang, 2018. "Effects of Item Calibration Errors on Computerized Adaptive Testing under Cognitive Diagnosis Models," Journal of Classification, Springer;The Classification Society, vol. 35(3), pages 437-465, October.
    4. Xuliang Gao & Daxun Wang & Yan Cai & Dongbo Tu, 2020. "Cognitive Diagnostic Computerized Adaptive Testing for Polytomously Scored Items," Journal of Classification, Springer;The Classification Society, vol. 37(3), pages 709-729, October.
    5. Hua-Hua Chang, 2015. "Psychometrics Behind Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 1-20, March.
    6. Edison M. Choe & Jinming Zhang & Hua-Hua Chang, 2018. "Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 650-673, September.
    7. Chenchen Ma & Jing Ouyang & Gongjun Xu, 2023. "Learning Latent and Hierarchical Structures in Cognitive Diagnosis Models," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 175-207, March.
    8. Xiao Li & Hanchen Xu & Jinming Zhang & Hua-hua Chang, 2023. "Deep Reinforcement Learning for Adaptive Learning Systems," Journal of Educational and Behavioral Statistics, , vol. 48(2), pages 220-243, April.
    9. Miguel A Sorrel & Juan R Barrada & Jimmy de la Torre & Francisco José Abad, 2020. "Adapting cognitive diagnosis computerized adaptive testing item selection rules to traditional item response theory," PLOS ONE, Public Library of Science, vol. 15(1), pages 1-17, January.
    10. Chia-Yi Chiu & Yuan-Pei Chang, 2021. "Advances in CD-CAT: The General Nonparametric Item Selection Method," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 1039-1057, December.
    11. Hong-Yun Liu & Xiao-Feng You & Wen-Yi Wang & Shu-Liang Ding & Hua-Hua Chang, 2013. "The Development of Computerized Adaptive Testing with Cognitive Diagnosis for an English Achievement Test in China," Journal of Classification, Springer;The Classification Society, vol. 30(2), pages 152-172, July.
    12. Shiyu Wang & Yinghan Chen, 2020. "Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models," Psychometrika, Springer;The Psychometric Society, vol. 85(3), pages 600-629, September.
    13. Ragip Terzi & Sedat Sen, 2019. "A Nondiagnostic Assessment for Diagnostic Purposes: Q-Matrix Validation and Item-Based Model Fit Evaluation for the TIMSS 2011 Assessment," SAGE Open, , vol. 9(1), pages 21582440198, February.
    14. Crabbe, Marjolein & Akinc, Deniz & Vandebroek, Martina, 2014. "Fast algorithms to generate individualized designs for the mixed logit choice model," Transportation Research Part B: Methodological, Elsevier, vol. 60(C), pages 1-15.
    15. Ping Chen & Tao Xin & Chun Wang & Hua-Hua Chang, 2012. "Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 201-222, April.
    16. Pasquale Anselmi & Egidio Robusto & Luca Stefanutti & Debora Chiusole, 2016. "An Upgrading Procedure for Adaptive Assessment of Knowledge," Psychometrika, Springer;The Psychometric Society, vol. 81(2), pages 461-482, June.
    17. Yu Wang & Chia-Yi Chiu & Hans Friedrich Köhn, 2023. "Nonparametric Classification Method for Multiple-Choice Items in Cognitive Diagnosis," Journal of Educational and Behavioral Statistics, , vol. 48(2), pages 189-219, April.
    18. Hiroshi Tamano & Daichi Mochihashi, 2023. "Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 487-526, June.
    19. Chun Wang & Hua-Hua Chang, 2011. "Item Selection in Multidimensional Computerized Adaptive Testing—Gaining Information from Different Angles," Psychometrika, Springer;The Psychometric Society, vol. 76(3), pages 363-384, July.
    20. Jingchen Liu & Zhiliang Ying & Stephanie Zhang, 2015. "A Rate Function Approach to Computerized Adaptive Testing for Cognitive Diagnosis," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 468-490, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:48:y:2023:i:6:p:842-865. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.