How accurate is the accuracy ratio in credit risk model validation?

The receiver operating curve and the cumulative accuracy profile visualize the ability of a credit scoring model to distinguish defaulting from nondefaulting counterparties. These curves lead to performance metrics such as the accuracy ratio and the area under the curve. Since these performance metrics are sample properties, we cannot draw firm conclusions on the model performance without knowing the sampling distribution or the sample variance. We present four methods to estimate the sample variance of the accuracy ratio and the area under the curve. The first method is based on numerical integration, the second and third methods assume specific score distributions, and the fourth method uses a correlation, leading to a distribution-independent equation for the sample variance. We demonstrate by simulations that the first method gives the best estimation of the sample variance. The distribution-independent equation gives reasonable estimations of the sample variance, but ignores higher-order effects that are distribution dependent.