IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2204.07989.html
   My bibliography  Save this paper

Second-order accuracy metrics for scoring models and their practical use

Author

Listed:
  • M. V. Pomazanov

Abstract

The paper proposes new second-order accuracy metrics for scoring or rating models, which show the target preference of the model, it is better to diagnose good objects or better to diagnose bad ones for a constant generally accepted predictive power determined by the first order metric that is known as the Gini index. There are two metrics, they have both an integral representation and a numerical one. The numerical representation of metrics is of two types, the first of which is based on binary events to evaluate the model, the second on the default probability given by the model. Comparison of the results of calculating the metrics allows you to validate the calibration settings of the scoring or rating model and reveals its distortions. The article provides examples of calculating second-order accuracy metrics for ratings of several rating agencies, as well as for the well known approach to calibration based on van der Burg's ROC curves.

Suggested Citation

  • M. V. Pomazanov, 2022. "Second-order accuracy metrics for scoring models and their practical use," Papers 2204.07989, arXiv.org, revised Nov 2022.
    • Handle: RePEc:arx:papers:2204.07989
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2204.07989
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Engelmann, Bernd & Hayden, Evelyn & Tasche, Dirk, 2003. "Measuring the Discriminative Power of Rating Systems," Discussion Paper Series 2: Banking and Financial Studies 2003,01, Deutsche Bundesbank.
    2. Dirk Tasche, 2009. "Estimating discriminatory power and PD curves when the number of defaults is small," Papers 0905.3928, arXiv.org, revised Mar 2010.
    3. Moon, Choon-Geol & Stotsky, Janet G, 1993. "Testing the Differences between the Determinants of Moody's and Standard & Poor's Ratings: An Application of Smooth Simulated Maximum Likelihood Estimation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 8(1), pages 51-69, Jan.-Marc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Henry Penikas, 2023. "Unaccounted model risk for Basel IRB models deemed acceptable by conventional validation criteria," Risk Management, Palgrave Macmillan, vol. 25(4), pages 1-25, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lukasz Prorokowski, 2016. "Rank-order statistics for validating discriminative power of credit risk models," Bank i Kredyt, Narodowy Bank Polski, vol. 47(3), pages 227-250.
    2. Wosnitza, Jan Henrik, 2022. "Calibration alternatives to logistic regression and their potential for transferring the dispersion of discriminatory power into uncertainties of probabilities of default," Discussion Papers 04/2022, Deutsche Bundesbank.
    3. Kristóf, Tamás, 2008. "A csődelőrejelzés és a nem fizetési valószínűség számításának módszertani kérdéseiről [Some methodological questions of bankruptcy prediction and probability of default estimation]," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(5), pages 441-461.
    4. Alexandr Karminsky & Anatoly Peresetsky, 2009. "Ratings as Measure of Financial Risk: Evolution, Function and Usage," Journal of the New Economic Association, New Economic Association, issue 1-2, pages 86-102.
    5. Kish, Richard J. & Hogan, Karen M. & Olson, Gerard, 1999. "Does the market perceive a difference in rating agencies?," The Quarterly Review of Economics and Finance, Elsevier, vol. 39(3), pages 363-377.
    6. Dirk Tasche, 2015. "Fitting a distribution to Value-at-Risk and Expected Shortfall, with an application to covered bonds," Papers 1505.07484, arXiv.org, revised Nov 2015.
    7. Dean Fantazzini & Silvia Figini, 2009. "Random Survival Forests Models for SME Credit Risk Measurement," Methodology and Computing in Applied Probability, Springer, vol. 11(1), pages 29-45, March.
    8. Shen, Chung-Hua & Huang, Yu-Li & Hasan, Iftekhar, 2012. "Asymmetric benchmarking in bank credit rating," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 22(1), pages 171-193.
    9. Hang Luo & Linfeng Chen, 2019. "Bond yield and credit rating: evidence of Chinese local government financing vehicles," Review of Quantitative Finance and Accounting, Springer, vol. 52(3), pages 737-758, April.
    10. Bissoondoyal-Bheenick, Emawtee, 2005. "An analysis of the determinants of sovereign ratings," Global Finance Journal, Elsevier, vol. 15(3), pages 251-280, February.
    11. Moro Russ A. & Härdle Wolfgang K. & Schäfer Dorothea, 2017. "Company rating with support vector machines," Statistics & Risk Modeling, De Gruyter, vol. 34(1-2), pages 55-67, June.
    12. En-Der Su & Shih-Ming Huang, 2010. "Comparing Firm Failure Predictions Between Logit, KMV, and ZPP Models: Evidence from Taiwan’s Electronics Industry," Asia-Pacific Financial Markets, Springer;Japanese Association of Financial Economics and Engineering, vol. 17(3), pages 209-239, September.
    13. Ramasubramanian Sundararajan & Tarun Bhaskar & Abhinanda Sarkar & Sridhar Dasaratha & Debasis Bal & Jayanth K. Marasanapalle & Beata Zmudzka & Karolina Bak, 2011. "Marketing Optimization in Retail Banking," Interfaces, INFORMS, vol. 41(5), pages 485-505, October.
    14. Tasche, Dirk, 2013. "Bayesian estimation of probabilities of default for low default portfolios," Journal of Risk Management in Financial Institutions, Henry Stewart Publications, vol. 6(3), pages 302-326, July.
    15. Georgiou, K. & Domazakis, G.N. & Pappas, D. & Yannacopoulos, A.N., 2021. "Markov chain lumpability and applications to credit risk modelling in compliance with the International Financial Reporting Standard 9 framework," European Journal of Operational Research, Elsevier, vol. 292(3), pages 1146-1164.
    16. Ralf Elsas & Sabine Mielert, 2010. "Unternehmenskrisen und der Wirtschaftsfonds Deutschland," Schmalenbach Journal of Business Research, Springer, vol. 62(61), pages 18-37, January.
    17. Bissoondoyal-Bheenick, Emawtee & Brooks, Robert & Yip, Angela Y.N., 2006. "Determinants of sovereign ratings: A comparison of case-based reasoning and ordered probit approaches," Global Finance Journal, Elsevier, vol. 17(1), pages 136-154, September.
    18. Rafael Repullo & Jesús Saurina & Carlos Trucharte, 2010. "Mitigating the pro-cyclicality of Basel II [Bank loan loss provisions: a re-examination of capital management, earnings management and signalling effects]," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 25(64), pages 659-702.
    19. Dierkes, Maik & Erner, Carsten & Langer, Thomas & Norden, Lars, 2013. "Business credit information sharing and default risk of private firms," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 2867-2878.
    20. Laura Auria & Rouslan A. Moro, 2008. "Support Vector Machines (SVM) as a Technique for Solvency Analysis," Discussion Papers of DIW Berlin 811, DIW Berlin, German Institute for Economic Research.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2204.07989. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.