Imbalance-Aware Evaluation and Hyperparameter Optimization of Supervised Machine Learning Models for Credit Card Fraud Detection

The financial sector is one of the industries where credit card fraud detection is a critical issue because the number of legitimate transactions is by far outnumbered by the number of fraud transactions carried out. This paper performs an imbalance-sensitive analysis and hyperparameter optimization of three supervised machine learning (SML) models (Logistic Regression, Random Forest, and XGBoost) on the European credit card fraud dataset (n = 284,807; fraud rate = 0.172%). It embraced the CRISP-DM process model as the data lifecycle model to guide it. The training partition was only subjected to SMOTE after 80/20 stratified split to avoid data leaking and the hyperparameters are optimized using stratified 3-fold cross-validation. Each tuned model was further probability threshold tuned with probability threshold set to 0.70 to maximize Precision-Recall operating point. All the experiments were executed in Google Colaboratory on Python 3.10. Precision, Recall, F1-Score, ROC-AUC and the Area Under the Precision-Recall Curve (AUPRC) were used to evaluate model performance, and AUPRC was chosen as the ultimate measure due to extreme imbalance in the classes. XGBoost developed as the most effective model in general, having the highest AUPRC (0.817), ROC-AUC (0.970) and the perfect combination of Precision = Recall = F1 = 0.81, which was achieved by tuning the probability threshold to 0.70. Random Forest had the best Precision (0.93) with AUPRC of 0.805 and hence it is the most appropriate model in the minimum false positive. Logistic Regression achieved maximum Recall (0.86) but had low Precision (0.10) which restricted its feasibility of operation even with threshold modification. These results indicate that XGBoost, together with SMOTE, systematic hyperparameter optimization, and threshold calibration, offers the best and balanced fraud detection at extreme imbalance in the classes.