IDEAS home Printed from https://ideas.repec.org/a/spr/jclass/v30y2013i2p251-275.html
   My bibliography  Save this article

Measuring the Reliability of Diagnostic Classification Model Examinee Estimates

Author

Listed:
  • Jonathan Templin
  • Laine Bradshaw

Abstract

Over the past decade, diagnostic classification models (DCMs) have become an active area of psychometric research. Despite their use, the reliability of examinee estimates in DCM applications has seldom been reported. In this paper, a reliability measure for the categorical latent variables of DCMs is defined. Using theory-and simulation-based results, we show how DCMs uniformly provide greater examinee estimate reliability than IRT models for tests of the same length, a result that is a consequence of the smaller range of latent variable values examinee estimates can take in DCMs. We demonstrate this result by comparing DCM and IRT reliability for a series of models estimated with data from an end-of-grade test, culminating with a discussion of how DCMs can be used to change the character of large scale testing, either by shortening tests that measure examinees unidimensionally or by providing more reliable multidimensional measurement for tests of the same length. Copyright Springer Science+Business Media New York 2013

Suggested Citation

  • Jonathan Templin & Laine Bradshaw, 2013. "Measuring the Reliability of Diagnostic Classification Model Examinee Estimates," Journal of Classification, Springer;The Classification Society, vol. 30(2), pages 251-275, July.
    • Handle: RePEc:spr:jclass:v:30:y:2013:i:2:p:251-275
    DOI: 10.1007/s00357-013-9129-4
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s00357-013-9129-4
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s00357-013-9129-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Maydeu-Olivares, Albert & Joe, Harry, 2005. "Limited- and Full-Information Estimation and Goodness-of-Fit Testing in 2n Contingency Tables: A Unified Framework," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 1009-1020, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jonathan Templin & Laine Bradshaw, 2014. "Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 317-339, April.
    2. Matthew S. Johnson & Sandip Sinharay, 2020. "The Reliability of the Posterior Probability of Skill Attainment in Diagnostic Classification Models," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 5-31, February.
    3. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.
    4. Shiyu Wang & Yan Yang & Steven Andrew Culpepper & Jeffrey A. Douglas, 2018. "Tracking Skill Acquisition With Cognitive Diagnosis Models: A Higher-Order, Hidden Markov Model With Covariates," Journal of Educational and Behavioral Statistics, , vol. 43(1), pages 57-87, February.
    5. Kazuhiro Yamaguchi & Kensuke Okada, 2020. "Variational Bayes Inference Algorithm for the Saturated Diagnostic Classification Model," Psychometrika, Springer;The Psychometric Society, vol. 85(4), pages 973-995, December.
    6. Matthew J. Madison & Laine P. Bradshaw, 2018. "Assessing Growth in a Diagnostic Classification Model Framework," Psychometrika, Springer;The Psychometric Society, vol. 83(4), pages 963-990, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alberto Maydeu-Olivares & Rosa Montaño, 2013. "How Should We Assess the Fit of Rasch-Type Models? Approximating the Power of Goodness-of-Fit Statistics in Categorical Data Analysis," Psychometrika, Springer;The Psychometric Society, vol. 78(1), pages 116-133, January.
    2. Vassilis Vasdekis & Silvia Cagnone & Irini Moustaki, 2012. "A Composite Likelihood Inference in Latent Variable Models for Ordinal Longitudinal Responses," Psychometrika, Springer;The Psychometric Society, vol. 77(3), pages 425-441, July.
    3. Chen, Yunxiao & Moustaki, Irini & Zhang, H, 2020. "A note on likelihood ratio tests for models with latent variables," LSE Research Online Documents on Economics 107490, London School of Economics and Political Science, LSE Library.
    4. Shaobo Jin & Fan Yang-Wallentin, 2017. "Asymptotic Robustness Study of the Polychoric Correlation Estimation," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 67-85, March.
    5. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.
    6. Ick Hoon Jin & Minjeong Jeon, 2019. "A Doubly Latent Space Joint Model for Local Item and Person Dependence in the Analysis of Item Response Data," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 236-260, March.
    7. Li Cai, 2010. "A Two-Tier Full-Information Item Factor Analysis Model with Applications," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 581-612, December.
    8. Jochen Ranger & Kay Brauer, 2022. "On the Generalized S − X 2 –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization," Journal of Educational and Behavioral Statistics, , vol. 47(2), pages 202-230, April.
    9. Albert Maydeu-Olivares & Harry Joe, 2006. "Limited Information Goodness-of-fit Testing in Multidimensional Contingency Tables," Psychometrika, Springer;The Psychometric Society, vol. 71(4), pages 713-732, December.
    10. Kim, Sung-Ho & Choi, Hyemi & Lee, Sangjin, 2009. "Estimate-based goodness-of-fit test for large sparse multinomial distributions," Computational Statistics & Data Analysis, Elsevier, vol. 53(4), pages 1122-1131, February.
    11. Wu, Jianmin & Bentler, Peter M., 2013. "Limited information estimation in binary factor analysis: A review and extension," Computational Statistics & Data Analysis, Elsevier, vol. 57(1), pages 392-403.
    12. Salim Moussa, 2016. "A two-step item response theory procedure for a better measurement of marketing constructs," Journal of Marketing Analytics, Palgrave Macmillan, vol. 4(1), pages 28-50, March.
    13. Yanlou Liu & Wei Tian & Tao Xin, 2016. "An Application of M2 Statistic to Evaluate the Fit of Cognitive Diagnostic Models," Journal of Educational and Behavioral Statistics, , vol. 41(1), pages 3-26, February.
    14. Qian, Zhiguang & Shapiro, Alexander, 2006. "Simulation-based approach to estimation of latent variable models," Computational Statistics & Data Analysis, Elsevier, vol. 51(2), pages 1243-1259, November.
    15. Jonathan Templin & Laine Bradshaw, 2014. "Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 317-339, April.
    16. Yuqi Gu & Jingchen Liu & Gongjun Xu & Zhiliang Ying, 2018. "Hypothesis Testing of the Q-matrix," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 515-537, September.
    17. Tang, Min & Slud, Eric V. & Pfeiffer, Ruth M., 2014. "Goodness of fit tests for linear mixed models," Journal of Multivariate Analysis, Elsevier, vol. 130(C), pages 176-193.
    18. Harry Joe & Alberto Maydeu-Olivares, 2010. "A General Family of Limited Information Goodness-of-Fit Statistics for Multinomial Data," Psychometrika, Springer;The Psychometric Society, vol. 75(3), pages 393-419, September.
    19. Genest Christian & Puccetti Giovanni, 2018. "A Journey Beyond The Gaussian World: An interview with Harry Joe," Dependence Modeling, De Gruyter, vol. 6(1), pages 288-297, December.
    20. Carmen Köhler & Alexander Robitzsch & Johannes Hartig, 2020. "A Bias-Corrected RMSD Item Fit Statistic: An Evaluation and Comparison to Alternatives," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 251-273, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jclass:v:30:y:2013:i:2:p:251-275. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.