IDEAS home Printed from https://ideas.repec.org/a/spr/advdac/v17y2023i4d10.1007_s11634-022-00528-0.html
   My bibliography  Save this article

A power-controlled reliability assessment for multi-class probabilistic classifiers

Author

Listed:
  • Hyukjun Gweon

    (Western University)

Abstract

In multi-class classification, the output of a probabilistic classifier is a probability distribution of the classes. In this work, we focus on a statistical assessment of the reliability of probabilistic classifiers for multi-class problems. Our approach generates a Pearson $$\chi ^2$$ χ 2 statistic based on the k-nearest-neighbors in the prediction space. Further, we develop a Bayesian approach for estimating the expected power of the reliability test that can be used for an appropriate sample size k. We propose a sampling algorithm and demonstrate that this algorithm obtains a valid prior distribution. The effectiveness of the proposed reliability test and expected power is evaluated through a simulation study. We also provide illustrative examples of the proposed methods with practical applications.

Suggested Citation

  • Hyukjun Gweon, 2023. "A power-controlled reliability assessment for multi-class probabilistic classifiers," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 17(4), pages 927-949, December.
    • Handle: RePEc:spr:advdac:v:17:y:2023:i:4:d:10.1007_s11634-022-00528-0
    DOI: 10.1007/s11634-022-00528-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11634-022-00528-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11634-022-00528-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Cheng, Dunlei & Branscum, Adam J. & Stamey, James D., 2010. "A Bayesian approach to sample size determination for studies designed to evaluate continuous medical tests," Computational Statistics & Data Analysis, Elsevier, vol. 54(2), pages 298-307, February.
    2. Rauch, G. & Kieser, M., 2013. "An expected power approach for the assessment of composite endpoints and their components," Computational Statistics & Data Analysis, Elsevier, vol. 60(C), pages 111-122.
    3. Allan H. Murphy & Robert L. Winkler, 1977. "Reliability of Subjective Probability Forecasts of Precipitation and Temperature," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 26(1), pages 41-47, March.
    4. J. A. Hartigan & M. A. Wong, 1979. "A K‐Means Clustering Algorithm," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 28(1), pages 100-108, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    2. Kolos Cs. Ágoston & Marianna E.-Nagy, 2024. "Mixed integer linear programming formulation for K-means clustering problem," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 32(1), pages 11-27, March.
    3. Zhang, Weibin & Zha, Huazhu & Zhang, Shuai & Ma, Lei, 2023. "Road section traffic flow prediction method based on the traffic factor state network," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 618(C).
    4. Jelle R Dalenberg & Luca Nanetti & Remco J Renken & René A de Wijk & Gert J ter Horst, 2014. "Dealing with Consumer Differences in Liking during Repeated Exposure to Food; Typical Dynamics in Rating Behavior," PLOS ONE, Public Library of Science, vol. 9(3), pages 1-11, March.
    5. Custodio João, Igor & Lucas, André & Schaumburg, Julia & Schwaab, Bernd, 2023. "Dynamic clustering of multivariate panel data," Journal of Econometrics, Elsevier, vol. 237(2).
    6. Gnidchenko, A., 2025. "World trade concentration and product market segregation," Journal of the New Economic Association, New Economic Association, vol. 66(1), pages 36-53.
    7. Thomas Astebro & Frank Fossen & Cédric Gutierrez, 2024. "Entrepreneurs: Clueless, Biased, Poor Heuristics, or Bayesian Machines?," Working Papers hal-04759301, HAL.
    8. Utkarsh J. Dang & Michael P.B. Gallaugher & Ryan P. Browne & Paul D. McNicholas, 2023. "Model-Based Clustering and Classification Using Mixtures of Multivariate Skewed Power Exponential Distributions," Journal of Classification, Springer;The Classification Society, vol. 40(1), pages 145-167, April.
    9. Bernd Scherer & Diogo Judice & Stephan Kessler, 2010. "Price reversals in global equity markets," Journal of Asset Management, Palgrave Macmillan, vol. 11(5), pages 332-345, December.
    10. Ugofilippo Basellini & Carlo Giovanni Camarda, 2020. "Modelling COVID-19 mortality at the regional level in Italy," Working Papers axq0sudakgkzhr-blecv, French Institute for Demographic Studies.
    11. D. Marc Kilgour & Yigal Gerchak, 2004. "Elicitation of Probabilities Using Competitive Scoring Rules," Decision Analysis, INFORMS, vol. 1(2), pages 108-113, June.
    12. Jing Xiao & Qiongqiong Xu & Chuanli Wu & Yuexia Gao & Tianqi Hua & Chenwu Xu, 2016. "Performance Evaluation of Missing-Value Imputation Clustering Based on a Multivariate Gaussian Mixture Model," PLOS ONE, Public Library of Science, vol. 11(8), pages 1-14, August.
    13. James K. Hammitt & Alexander I. Shlyakhter, 1999. "The Expected Value of Information and the Probability of Surprise," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 135-152, February.
    14. Humberto Elias Garcia Lopes & Marlusa de Sevilha Gosling, 2021. "Cluster Analysis in Practice: Dealing with Outliers in Managerial Research," RAC - Revista de Administração Contemporânea (Journal of Contemporary Administration), ANPAD - Associação Nacional de Pós-Graduação e Pesquisa em Administração, vol. 25(1), pages 200081-2000.
    15. Mónika-Anetta Alt & Vizeli Ibolya, 2021. "Identifying Relevant Segments of Potential Banking Chatbot Users Based on Technology Adoption Behavior," Tržište/Market, Faculty of Economics and Business, University of Zagreb, vol. 33(2), pages 165-183.
    16. Naderi, Mehrdad & Hung, Wen-Liang & Lin, Tsung-I & Jamalizadeh, Ahad, 2019. "A novel mixture model using the multivariate normal mean–variance mixture of Birnbaum–Saunders distributions and its application to extrasolar planets," Journal of Multivariate Analysis, Elsevier, vol. 171(C), pages 126-138.
    17. Alan Fernihough & Kevin Hjortshøj, 2021. "Coal and the European Industrial Revolution," The Economic Journal, Royal Economic Society, vol. 131(635), pages 1135-1149.
    18. Dimitriadis, Timo & Gneiting, Tilmann & Jordan, Alexander I. & Vogel, Peter, 2024. "Evaluating probabilistic classifiers: The triptych," International Journal of Forecasting, Elsevier, vol. 40(3), pages 1101-1122.
    19. Mohiuddin, Hossain & Fitch-Polse, Dillon T. & Handy, Susan L., 2024. "Examining market segmentation to increase bike-share use and enhance equity: The case of the greater Sacramento region," Transport Policy, Elsevier, vol. 145(C), pages 279-290.
    20. David Bessler & Robert Ruffley, 2004. "Prequential analysis of stock market returns," Applied Economics, Taylor & Francis Journals, vol. 36(5), pages 399-412.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:advdac:v:17:y:2023:i:4:d:10.1007_s11634-022-00528-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.