Forecasting Out-of-Time Credit Scoring Model Risk

This paper addresses the challenge of forecasting the best-performing credit scoring model in outof-time settings, focusing on the decision between segmented (bank-specific) and full data (financial system-wide) models. Building upon the Credit Scoring Model Risk (CSMR) metric, defined as one minus the correlation between observed defaults and predicted scores, we highlight the instability of in-sample performance measures when applied to evolving loan portfolios and changing macroeconomic conditions. We propose three complementary approaches to predict out-of-time model performance: (i) an analytical method based on Copas shrinkage concept utilizing estimated covariances and prediction variances; (ii) a Monte Carlo simulation leveraging average model predictions to simulate default events; and (iii) a Bayesian estimation framework for covariances grounded in conditional expectations of predictions given default. Empirical analysis using a large Brazilian loan dataset reveals that segmented models outperform full data models in in-sample contexts but not consistently out-of-time. Among the approaches, the Monte Carlo simulation achieved the highest accuracy (70.8%) in forecasting the superior out-of-time model, followed by the Bayesian method (66.7%) and the analytical shrinkage approach (54.2%). The study underscores the importance of considering population shifts via the Population Stability Index (PSI) to detect model decalibration and overfitting. The proposed methodologies offer practitioners and regulators practical tools for informed model selection, enhancing predictive reliability over time amid portfolio and economic dynamics.