IDEAS home Printed from https://ideas.repec.org/a/spr/advdac/v17y2023i4d10.1007_s11634-022-00528-0.html
   My bibliography  Save this article

A power-controlled reliability assessment for multi-class probabilistic classifiers

Author

Listed:
  • Hyukjun Gweon

    (Western University)

Abstract

In multi-class classification, the output of a probabilistic classifier is a probability distribution of the classes. In this work, we focus on a statistical assessment of the reliability of probabilistic classifiers for multi-class problems. Our approach generates a Pearson $$\chi ^2$$ χ 2 statistic based on the k-nearest-neighbors in the prediction space. Further, we develop a Bayesian approach for estimating the expected power of the reliability test that can be used for an appropriate sample size k. We propose a sampling algorithm and demonstrate that this algorithm obtains a valid prior distribution. The effectiveness of the proposed reliability test and expected power is evaluated through a simulation study. We also provide illustrative examples of the proposed methods with practical applications.

Suggested Citation

  • Hyukjun Gweon, 2023. "A power-controlled reliability assessment for multi-class probabilistic classifiers," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 17(4), pages 927-949, December.
    • Handle: RePEc:spr:advdac:v:17:y:2023:i:4:d:10.1007_s11634-022-00528-0
    DOI: 10.1007/s11634-022-00528-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11634-022-00528-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11634-022-00528-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Cheng, Dunlei & Branscum, Adam J. & Stamey, James D., 2010. "A Bayesian approach to sample size determination for studies designed to evaluate continuous medical tests," Computational Statistics & Data Analysis, Elsevier, vol. 54(2), pages 298-307, February.
    2. Rauch, G. & Kieser, M., 2013. "An expected power approach for the assessment of composite endpoints and their components," Computational Statistics & Data Analysis, Elsevier, vol. 60(C), pages 111-122.
    3. Allan H. Murphy & Robert L. Winkler, 1977. "Reliability of Subjective Probability Forecasts of Precipitation and Temperature," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 26(1), pages 41-47, March.
    4. J. A. Hartigan & M. A. Wong, 1979. "A K‐Means Clustering Algorithm," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 28(1), pages 100-108, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rauch, G. & Kieser, M., 2013. "An expected power approach for the assessment of composite endpoints and their components," Computational Statistics & Data Analysis, Elsevier, vol. 60(C), pages 111-122.
    2. Xu, Jing & Wang, Xiaoying & Gu, Yujiong & Ma, Suxia, 2023. "A data-based day-ahead scheduling optimization approach for regional integrated energy systems with varying operating conditions," Energy, Elsevier, vol. 283(C).
    3. Carlos Carrasco-Farré, 2022. "The fingerprints of misinformation: how deceptive content differs from reliable sources in terms of cognitive effort and appeal to emotions," Palgrave Communications, Palgrave Macmillan, vol. 9(1), pages 1-18, December.
    4. Felix Mbuga & Cristina Tortora, 2021. "Spectral Clustering of Mixed-Type Data," Stats, MDPI, vol. 5(1), pages 1-11, December.
    5. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    6. Kolos Cs. Ágoston & Marianna E.-Nagy, 2024. "Mixed integer linear programming formulation for K-means clustering problem," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 32(1), pages 11-27, March.
    7. Zhang, Weibin & Zha, Huazhu & Zhang, Shuai & Ma, Lei, 2023. "Road section traffic flow prediction method based on the traffic factor state network," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 618(C).
    8. K J Wilson & M Farrow, 2010. "Bayes linear kinematics in the analysis of failure rates and failure time distributions," Journal of Risk and Reliability, , vol. 224(4), pages 309-321, December.
    9. Michal Bernardelli & Zbigniew Korzeb & Pawel Niedziolka, 2021. "The banking sector as the absorber of the COVID-19 crisis’ economic consequences: perception of WSE investors," Oeconomia Copernicana, Institute of Economic Research, vol. 12(2), pages 335-374, June.
    10. Jelle R Dalenberg & Luca Nanetti & Remco J Renken & René A de Wijk & Gert J ter Horst, 2014. "Dealing with Consumer Differences in Liking during Repeated Exposure to Food; Typical Dynamics in Rating Behavior," PLOS ONE, Public Library of Science, vol. 9(3), pages 1-11, March.
    11. Custodio João, Igor & Lucas, André & Schaumburg, Julia & Schwaab, Bernd, 2023. "Dynamic clustering of multivariate panel data," Journal of Econometrics, Elsevier, vol. 237(2).
    12. Carlos Fernández-Hernández & Carmelo J. León & Jorge E. Araña & Flora Díaz-Pére, 2016. "Market segmentation, activities and environmental behaviour in rural tourism," Tourism Economics, , vol. 22(5), pages 1033-1054, October.
    13. Gnidchenko, A., 2025. "World trade concentration and product market segregation," Journal of the New Economic Association, New Economic Association, vol. 66(1), pages 36-53.
    14. Hafid Kadi & Mohammed Rebbah & Boudjelal Meftah & Olivier Lézoray, 2021. "A Data Representation Model for Personalized Medicine," International Journal of Healthcare Information Systems and Informatics (IJHISI), IGI Global, vol. 16(4), pages 1-25, October.
    15. Thomas Astebro & Frank Fossen & Cédric Gutierrez, 2024. "Entrepreneurs: Clueless, Biased, Poor Heuristics, or Bayesian Machines?," Working Papers hal-04759301, HAL.
    16. Zhang, Tonglin & Lin, Ge, 2021. "Generalized k-means in GLMs with applications to the outbreak of COVID-19 in the United States," Computational Statistics & Data Analysis, Elsevier, vol. 159(C).
    17. Hung Trong Hoang & Phuong Thao Nguyen & Nam Cong-Nhat Huynh & Tam Thi-Thanh Nguyen & Trang Thi Huyen Tu & Michael George Botelho & Lan Van Nguyen & Kaori Shima & Tomonori Sasahira, 2023. "Reliability of online dental final exams in the pre and post COVID-19 era: A comparative study," PLOS ONE, Public Library of Science, vol. 18(5), pages 1-13, May.
    18. Andreas Lackner & Michael Müller & Magdalena Gamperl & Delyana Stoeva & Olivia Langmann & Henrieta Papuchova & Elisabeth Roitinger & Gerhard Dürnberger & Richard Imre & Karl Mechtler & Paulina A. Lato, 2023. "The Fgf/Erf/NCoR1/2 repressive axis controls trophoblast cell fate," Nature Communications, Nature, vol. 14(1), pages 1-20, December.
    19. Utkarsh J. Dang & Michael P.B. Gallaugher & Ryan P. Browne & Paul D. McNicholas, 2023. "Model-Based Clustering and Classification Using Mixtures of Multivariate Skewed Power Exponential Distributions," Journal of Classification, Springer;The Classification Society, vol. 40(1), pages 145-167, April.
    20. Beibei Yu & Zhonghui Wang & Haowei Mu & Li Sun & Fengning Hu, 2019. "Identification of Urban Functional Regions Based on Floating Car Track Data and POI Data," Sustainability, MDPI, vol. 11(23), pages 1-18, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:advdac:v:17:y:2023:i:4:d:10.1007_s11634-022-00528-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.