Probabilistic classification with discriminative and generative models: credit-scoring application

This study investigates the potential of probabilistic classification to enhance credit-scoring accuracy, with a focus on model validation through reliability thresholds. By quantifying prediction confidence as a risk validation metric, the proposed framework provides a robust tool for assessing model performance under various reliability criteria, addressing key challenges in credit risk model validation. A comparison of discriminative (random forest, logistic regression) and generative (probabilistic neural network, naive Bayes) models is conducted to determine if leveraging the reliability of classifier predictions can improve their overall performance. The class probability values generated by these models are analyzed as a measure of prediction confidence (reliability level). It is hypothesized that increasing the reliability threshold (ie, requiring higher class probability values for predictions to be considered) can reduce the number of predictions while improving their accuracy. The findings support this hypothesis for most models. Both discriminative and generative approaches demonstrate increased accuracy with higher reliability thresholds. While inconsistent results are observed for the naive Bayes model, the other models exhibit comparable performance, suggesting that the model’s base performance is more influential than its discriminative or generative nature. These findings highlight the potential of incorporating prediction reliability thresholds as a practical approach to risk model validation in credit-scoring contexts.