Development and external evaluation of an interpretable machine-learning model for early prediction of organ failure in higher-risk acute pancreatitis patients: A multicentre cohort study

Organ failure (OF) is the most important determinant of prognosis in acute pancreatitis (AP) and its duration defines disease severity. Early identification of individuals at high risk of developing OF is crucial. We aimed to develop machine-learning models, benchmarked against a feedforward multilayer-perceptron (MLP) neural network, to predict new-onset OF at admission using static admission-day variables. In this study, data were extracted from MIMIC-IV (development cohort) and a multicentre AP cohort from two Chinese tertiary teaching hospitals (evaluation cohort), which excluded mild AP and therefore comprised patients requiring ICU-level or high-dependency care. Features were selected with Boruta and LASSO. Five machine-learning models and one multilayer-perceptron benchmark were developed in MIMIC-IV and externally evaluated in the Xiangya cohort without updating. SHapley Additive exPlanations (SHAP) were used to visualize decision-making patterns and individual prediction interpretations. The best-performing model was deployed as an interactive, web-based tool. We found that in the discovery cohort, 341 (23.9%) of 1429 AP patients developed new-onset OF within 28 days of admission while 45 of 216 patients (20.8%) in the validation cohort developed OF. Boruta and LASSO algorithms identified six key predictors including blood urea nitrogen, platelets, triglyceride-glucose index, albumin, white blood cells, and partial thromboplastin time, which were used to construct the ML and MLP models. XGBoost gave the best discrimination on external evaluation (AUC 0.837, 95% CI 0.771–0.902). Logistic recalibration improved calibration, and the recalibrated XGBoost model was implemented as a web-based research prototype (Decent app). In conclusion, XGBoost predicted new-onset OF with good discrimination in a severity-enriched AP cohort, and SHAP made individual predictions interpretable. Model selection, threshold selection and recalibration all used the evaluation cohort, so the deployed model still requires an independent cohort. Prospective usability and clinical-impact studies are needed before clinical use.Author summary: Acute pancreatitis (AP) is a serious inflammatory condition where the development of organ failure (OF) is the most critical factor determining patient outcomes. Early identification of patients at high risk for OF is essential for improving care, but accurate prediction at the time of hospital admission remains a challenge. In this study, we developed machine-learning models, benchmarked against a multilayer perceptron, and evaluated them externally to predict new-onset OF in AP patients on admission to hospital. Using data from large international cohorts, we identified six key clinical characteristics at admission, including blood urea nitrogen, platelets, triglyceride-glucose index, albumin, white blood cells, and partial thromboplastin time levels, to train our models. Among the algorithms tested on admission-day tabular data, XGBoost performed best. The cohort used for evaluation excluded mild AP, so our results apply to patients needing ICU-level, high-dependency or close monitoring, not to unselected admissions. We then recalibrated the XGBoost model in the same cohort and built a web application (Decent app) that returns a risk estimate with an explanation of the factors behind it. Because the recalibration and the risk threshold came from the cohort in which we assessed them, the application has not been independently validated. It is a research prototype for risk calculation rather than a decision-support tool, and prospective usability, workflow and clinical-impact studies are needed first.