Aspiring to clinical significance: Insights from developing and evaluating a machine learning model to predict emergency department return visit admissions

Return visit admissions (RVA), which are instances where patients discharged from the emergency department (ED) rapidly return and require hospital admission, have been associated with quality issues and adverse outcomes. We developed and validated a machine learning model to predict 72-hour RVA using electronic health records (EHR) data. Study data were extracted from EHR data in 2019 from three urban EDs. The development and independent validation datasets included 62,154 patients from two EDs and 73,453 patients from one ED, respectively. Multiple machine learning algorithms were evaluated, including deep significance clustering (DICE), regularized logistic regression (LR), Gradient Boosting Decision Tree, and XGBoost. These machine learning models were also compared against an existing clinical risk score. To support clinical actionability, clinician investigators conducted manual chart reviews of the cases identified by the model. Chart reviews categorized predicted cases across index ED discharge diagnosis and RVA root cause classifications. The best-performing model achieved an AUC of 0.87 in the development site (test set) and 0.75 in the independent validation set. The model, which combined DICE and LR, boosted predictive performance while providing well-defined features. The model was relatively robust to sensitivity analyses regarding performance across age, race, and by varying predictor availability but less robust across diagnostic groups. Clinician examination demonstrated discrete model performance characteristics within clinical subtypes of RVA. This machine learning model demonstrated a strong predictive performance for 72- RVA. Despite the limited clinical actionability potentially due to model complexity, the rarity of the outcome, and variable relevance, the clinical examination offered guidance on further variable inclusion for enhanced predictive accuracy and actionability.Author summary: This study developed a model to predict emergency department (ED) return visit admissions (RVA) which are clinical events in which patients discharged from the ED rapidly return and require hospital admission. RVA are multifactorial but have been associated with preventable adverse outcomes. We developed a predictive model by evaluating several machine learning techniques and compared this to an existing clinical score. The model drew data from electronic health records at three urban EDs. Clinicians conducted a clinical evaluation of the model input and output through manual chart reviews. The model was found to be predictive and fairly generalizable but lacked clinical actionability. Findings will inform future model development process by ensuring that we include variables are that predictive and explainable to clinicians in a way that leads to actionability.