IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i9p1493-d806515.html
   My bibliography  Save this article

The Utility of Receiver Operating Characteristic Curve in Educational Assessment: Performance Prediction

Author

Listed:
  • Hyunsuk Han

    (Department of Counseling Psychology, Kyungil University, Gyeongsan 38428, Korea)

Abstract

When examinees are classified into groups based on scores from educational assessment, two indices are widely used to gauge the psychometric quality of the classifications: accuracy and consistency. The two indices take correct classifications into consideration while overlooking incorrect ones, where unbalanced class distribution threatens the validity of results from the accuracy and consistency indices. The single values produced from the two indices also fail to address the inconsistent accuracy of the classifier across different cut score locations. The current study proposed the concept of classification quality, which utilizes the receiver operating characteristics (ROC) graph to comprehensively evaluate the performance of classifiers. The ROC graph illustrates the tradeoff between benefits (true positive rate) and costs (false positive rate) in classification. In this article, a simulation study was conducted to demonstrate how to generate and interpret ROC graphs in educational assessment and the benefits of using ROC graphs to interpret classification quality. The results show that ROC graphs provide an efficient approach to (a) visualize the fluctuating performance of scoring classifiers, (b) address the unbalanced class distribution issue inherent in the accuracy and consistency indices, and (c) produce more accurate estimation of the classification results.

Suggested Citation

  • Hyunsuk Han, 2022. "The Utility of Receiver Operating Characteristic Curve in Educational Assessment: Performance Prediction," Mathematics, MDPI, vol. 10(9), pages 1-11, April.
    • Handle: RePEc:gam:jmathe:v:10:y:2022:i:9:p:1493-:d:806515
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/9/1493/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/10/9/1493/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Allen Fleishman, 1978. "A method for simulating non-normal distributions," Psychometrika, Springer;The Psychometric Society, vol. 43(4), pages 521-532, December.
    2. Huynh Huynh, 1976. "Statistical consideration of mastery scores," Psychometrika, Springer;The Psychometric Society, vol. 41(1), pages 65-78, March.
    3. Jun Zhang & Shane Mueller, 2005. "A note on ROC analysis and non-parametric estimate of sensitivity," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 203-212, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fabio Farella, 2025. "Equality of Opportunity and Efficiency in Tertiary Education: a Data-Driven Perspective," SERIES 02-2025, Dipartimento di Economia e Finanza - Università degli Studi di Bari "Aldo Moro", revised May 2025.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Kunpeng & Lin, Wei, 2024. "Threshold spatial autoregressive model," Journal of Econometrics, Elsevier, vol. 244(1).
    2. Schinckus, Christophe, 2015. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 35(C), pages 104-110.
    3. Max Auerswald & Morten Moshagen, 2015. "Generating Correlated, Non-normally Distributed Data Using a Non-linear Structural Model," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 920-937, December.
    4. Mohan D. Pant & Todd C. Headrick, 2017. "Simulating Uniform- and Triangular- Based Double Power Method Distributions," Journal of Statistical and Econometric Methods, SCIENPRESS Ltd, vol. 6(1), pages 1-1.
    5. Huynh Huynh, 1980. "A nonrandomized minimax solution for passing scores in the binomial error model," Psychometrika, Springer;The Psychometric Society, vol. 45(2), pages 167-182, June.
    6. Schinckus, Christophe, 2018. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 45(C), pages 1-6.
    7. Huynh Huynh, 1982. "A bayesian procedure for mastery decisions based on multivariate normal test data," Psychometrika, Springer;The Psychometric Society, vol. 47(3), pages 309-319, September.
    8. Pelagatti, Matteo M. & Sen, Pranab K., 2013. "Rank tests for short memory stationarity," Journal of Econometrics, Elsevier, vol. 172(1), pages 90-105.
    9. Emanuela Raffinetti & Pier Alda Ferrari, 2021. "A dependence measure flow tree through Monte Carlo simulations," Quality & Quantity: International Journal of Methodology, Springer, vol. 55(2), pages 467-496, April.
    10. Yen Lee & David Kaplan, 2018. "Generating Multivariate Ordinal Data via Entropy Principles," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 156-181, March.
    11. M Hashem Pesaran & Takashi Yamagata, 2012. "Testing CAPM with a Large Number of Assets," Discussion Papers 12/05, Department of Economics, University of York.
    12. Beasley, T. Mark & Zumbo, Bruno D., 2003. "Comparison of aligned Friedman rank and parametric methods for testing interactions in split-plot designs," Computational Statistics & Data Analysis, Elsevier, vol. 42(4), pages 569-593, April.
    13. Irmer, Julien Patrick & Klein, Andreas & Schermelleh-Engel, Karin, 2024. "A General Model-Implied Simulation-Based Power Estimation Method for Correctly and Misspecfied Models: Applications to Nonlinear and Linear Structural Equation Models," OSF Preprints pe5bj, Center for Open Science.
    14. Fei Gu & Hao Wu, 2016. "Raw Data Maximum Likelihood Estimation for Common Principal Component Models: A State Space Approach," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 751-773, September.
    15. Pasquale Dolce & Cristina Davino & Domenico Vistocco, 2022. "Quantile composite-based path modeling: algorithms, properties and applications," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 16(4), pages 909-949, December.
    16. Peter Wludyka, 1999. "Two non-parametric, analysis-of-means-type tests for homogeneity of variances," Journal of Applied Statistics, Taylor & Francis Journals, vol. 26(2), pages 243-256.
    17. Yadira Pazmiño & José Juan de Felipe & Marc Vallbé & Franklin Cargua & Luis Quevedo, 2021. "Identification of a Set of Variables for the Classification of Páramo Soils Using a Nonparametric Model, Remote Sensing, and Organic Carbon," Sustainability, MDPI, vol. 13(16), pages 1-22, August.
    18. Po-Hsien Huang, 2017. "Asymptotics of AIC, BIC, and RMSEA for Model Selection in Structural Equation Modeling," Psychometrika, Springer;The Psychometric Society, vol. 82(2), pages 407-426, June.
    19. Ke-Hai Yuan & Peter Bentler, 2002. "On robusiness of the normal-theory based asymptotic distributions of three reliability coefficient estimates," Psychometrika, Springer;The Psychometric Society, vol. 67(2), pages 251-259, June.
    20. Vacca, Gianmarco & Zoia, Maria Grazia, 2019. "Kurtosis analysis in GARCH models with Gram–Charlier-like innovations," Economics Letters, Elsevier, vol. 183(C), pages 1-1.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:9:p:1493-:d:806515. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.