Evaluating fairness of machine learning prediction of prolonged wait times in Emergency Department with Interpretable eXtreme gradient boosting

It is essential to evaluate performance and assess quality before applying artificial intelligence (AI) and machine learning (ML) models to clinical practice. This study utilized ML to predict patient wait times in the Emergency Department (ED), determine model performance accuracies, and conduct fairness evaluations to further assess ethnic disparities in using ML for wait time prediction among different patient populations in the ED. This retrospective observational study included adult patients (age ≥18 years) in the ED (n=173,856 visits) who were assigned an Emergency Severity Index (ESI) level of 3 at triage. Prolonged wait time was defined as waiting time ≥30 minutes. We employed extreme gradient boosting (XGBoost) for predicting prolonged wait times. Model performance was assessed with accuracy, recall, precision, F1 score, and false negative rate (FNR). To perform the global and local interpretation of feature importance, we utilized Shapley additive explanations (SHAP) to interpret the output from the XGBoost model. Fairness in ML models were evaluated across sensitive attributes (sex, race and ethnicity, and insurance status) at both subgroup and individual levels. We found that nearly half (48.43%, 84,195) of ED patient visits demonstrated prolonged ED wait times. XGBoost model exhibited moderate accuracy performance (AUROC=0.81). When fairness was evaluated with FNRs, unfairness existed across different sensitive attributes (male vs. female, Hispanic vs. Non-Hispanic White, and patients with insurances vs. without insurance). The predicted FNRs were lower among females, Hispanics, and patients without insurance compared to their counterparts. Therefore, XGBoost model demonstrated acceptable performance in predicting prolonged wait times in ED visits. However, disparities arise in predicting patients with different sex, race and ethnicity, and insurance status. To enhance the utility of ML model predictions in clinical practice, conducting performance assessments and fairness evaluations are crucial.Author summary: We used machine learning to predict patient wait times in the Emergency Department (ED), evaluated the model’s performance, and conducted fairness assessments. We did a retrospective study including adult patients (age ≥18 years) in the ED (n=173,856 visits with 99,178 unique patients) who were assigned an Emergency Severity Index (ESI) level of 3 at triage. We defined prolonged wait time as a waiting time of ≥30 minutes, and we employed extreme gradient boosting (XGBoost) to predict these ED visits with prolonged wait times. Nearly half of the ED visits in our study experienced prolonged wait times. Our XGBoost model demonstrated moderate predictive accuracy. However, we observed disparities in the model’s performance across different demographic groups, including sex, race and ethnicity, and health insurance. To ensure the practical applicability of artificial intelligence and machine learning models in clinical settings, it is essential to perform both accuracy assessments and fairness evaluations.