IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v50y2025i4p604-631.html

Estimating the Reliability of Skill Transitions in Longitudinal Diagnostic Classification Models

Author

Listed:
  • Madeline A. Schellman

    (University of Georgia)

  • Matthew J. Madison

    (University of Georgia)

Abstract

Diagnostic classification models (DCMs) have grown in popularity as stakeholders increasingly desire actionable information related to students’ skill competencies. Longitudinal DCMs offer a psychometric framework for providing estimates of students’ proficiency status transitions over time. For both cross-sectional and longitudinal DCMs, it is important that researchers estimate and report reliability so stakeholders and end-users can evaluate the trustworthiness of results. Over the past decade, researchers have developed and applied various metrics for reliability in the DCM framework. This study extends these metrics onto the longitudinal DCM context and consists of three parts: (a) the theory and development of the new longitudinal DCM reliability metrics, (b) a simulation study to examine the performance of the developed metrics and establish thresholds, and (c) an empirical data analysis to illustrate an application of the developed metrics. This paper concludes with a discussion of our recommendations for applying the developed metrics.

Suggested Citation

  • Madeline A. Schellman & Matthew J. Madison, 2025. "Estimating the Reliability of Skill Transitions in Longitudinal Diagnostic Classification Models," Journal of Educational and Behavioral Statistics, , vol. 50(4), pages 604-631, August.
    • Handle: RePEc:sae:jedbes:v:50:y:2025:i:4:p:604-631
    DOI: 10.3102/10769986241256032
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986241256032
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986241256032?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Chun Wang, 2021. "Using Penalized EM Algorithm to Infer Learning Trajectories in Latent Transition CDM," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 167-189, March.
    2. Jonathan Templin & Laine Bradshaw, 2013. "Measuring the Reliability of Diagnostic Classification Model Examinee Estimates," Journal of Classification, Springer;The Classification Society, vol. 30(2), pages 251-275, July.
    3. Peida Zhan & Hong Jiao & Dandan Liao & Feiming Li, 2019. "A Longitudinal Higher-Order Diagnostic Classification Model," Journal of Educational and Behavioral Statistics, , vol. 44(3), pages 251-281, June.
    4. Matthew S. Johnson & Sandip Sinharay, 2020. "The Reliability of the Posterior Probability of Skill Attainment in Diagnostic Classification Models," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 5-31, February.
    5. Matthew J. Madison & Laine P. Bradshaw, 2018. "Assessing Growth in a Diagnostic Classification Model Framework," Psychometrika, Springer;The Psychometric Society, vol. 83(4), pages 963-990, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xin Xu & Shixiu Ren & Xintong Shan & Danhui Zhang, 2026. "A Polytomous Extension of the Higher-Order, Hidden Markov Model With Covariates and Hierarchical Learning Trajectories," Journal of Educational and Behavioral Statistics, , vol. 51(1), pages 206-247, February.
    2. Kazuhiro Yamaguchi, 2023. "Bayesian Analysis Methods for Two-Level Diagnosis Classification Models," Journal of Educational and Behavioral Statistics, , vol. 48(6), pages 773-809, December.
    3. Matthew S. Johnson & Sandip Sinharay, 2020. "The Reliability of the Posterior Probability of Skill Attainment in Diagnostic Classification Models," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 5-31, February.
    4. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.
    5. Kazuhiro Yamaguchi, 2025. "Bayesian Diagnostic Classification Models for a Partially Known Q-Matrix," Journal of Educational and Behavioral Statistics, , vol. 50(2), pages 331-382, April.
    6. Kazuhiro Yamaguchi & Kensuke Okada, 2020. "Variational Bayes Inference Algorithm for the Saturated Diagnostic Classification Model," Psychometrika, Springer;The Psychometric Society, vol. 85(4), pages 973-995, December.
    7. Yan Li & Chao Huang & Jia Liu, 2023. "Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?," Journal of Educational and Behavioral Statistics, , vol. 48(6), pages 842-865, December.
    8. Shiyu Wang & Yan Yang & Steven Andrew Culpepper & Jeffrey A. Douglas, 2018. "Tracking Skill Acquisition With Cognitive Diagnosis Models: A Higher-Order, Hidden Markov Model With Covariates," Journal of Educational and Behavioral Statistics, , vol. 43(1), pages 57-87, February.
    9. Muhamad Firdaus Mohd Noh & Mohd Effendi Ewan bin Mohd Matore, 2024. "Conundrum and Considerations in Cognitive Diagnostic Assessment for Language Proficiency Evaluation," Information Management and Business Review, AMH International, vol. 16(2), pages 63-72.
    10. Zhemin Zhu & Bing Jia, 2025. "Study on Mathematics Skill of Sixth-Grade Students in Rural China Based on Cognitive Diagnostic Models," SAGE Open, , vol. 15(4), pages 21582440251, December.
    11. Shiyu Wang & Yinghan Chen, 2020. "Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models," Psychometrika, Springer;The Psychometric Society, vol. 85(3), pages 600-629, September.
    12. Steven Andrew Culpepper, 2019. "An Exploratory Diagnostic Model for Ordinal Responses with Binary Attributes: Identifiability and Estimation," Psychometrika, Springer;The Psychometric Society, vol. 84(4), pages 921-940, December.
    13. Jonathan Templin & Laine Bradshaw, 2014. "Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 317-339, April.
    14. Matthew J. Madison & Laine P. Bradshaw, 2018. "Assessing Growth in a Diagnostic Classification Model Framework," Psychometrika, Springer;The Psychometric Society, vol. 83(4), pages 963-990, December.
    15. Kentaro Fukushima & Nao Uchida & Kensuke Okada, 2025. "Modeling Partial Knowledge in Multiple-Choice Cognitive Diagnostic Assessment," Journal of Educational and Behavioral Statistics, , vol. 50(1), pages 5-43, February.
    16. Hiroshi Tamano & Daichi Mochihashi, 2023. "Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 487-526, June.
    17. Qianru Liang & Jimmy de la Torre & Nancy Law, 2023. "Latent Transition Cognitive Diagnosis Model With Covariates: A Three-Step Approach," Journal of Educational and Behavioral Statistics, , vol. 48(6), pages 690-718, December.
    18. Peida Zhan & Xin Qiao, 2022. "DIAGNOSTIC Classification Analysis of Problem-Solving Competence using Process Data: An Item Expansion Method," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1529-1547, December.
    19. Yinghan Chen & Ying Liu & Steven Andrew Culpepper & Yuguo Chen, 2021. "Inferring the Number of Attributes for the Exploratory DINA Model," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 30-64, March.
    20. Steven Andrew Culpepper, 2023. "A Note on Weaker Conditions for Identifying Restricted Latent Class Models for Binary Responses," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 158-174, March.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:50:y:2025:i:4:p:604-631. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.