The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial

Reinforcement learning (RL) is a type of artificial intelligence for making optimal choices. In healthcare, researchers generally use offline RL (ORL), where models are trained and evaluated from retrospective observational data. To accommodate naturally irregular clinical records, researchers often resample the data into uniform time intervals before training (known as binning). However, binning may create counterfactual trajectories that do not represent clinical practice. We first quantified these distortions in patients receiving continuous insulin infusions in the MIMIC-IV intensive care database. Binning reversed the apparent causal order for over 70% of paired glucose measurements and insulin adjustments within the same interval, and frequently masked significant dysglycaemic events and merged distinct treatment decisions together. We then conducted an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. The simulator was modified to include stochastic intervals between decisions and used to generate a training dataset for offline RL. We trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals. When deployed back into the simulated environment, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data, with 4-hour binning causing all agents to perform worse than the dataset’s baseline. Retrospective evaluation on resampled data actively obscured this effect, predicting 1.5–3x better returns than agents achieved in practice. We recommend that future research in this area prioritises datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care.Author summary: Artificial Intelligence (AI) models could help future doctors by recommending personalised treatments for patients. These models are currently trained and tested using historical patient records rather than real patients. Medical decisions happen at unpredictable intervals, but before training an AI model, researchers usually shift and align the timings of recorded decisions into a regular schedule (‘temporal resampling’). This makes training the AI models easier with standard techniques. We investigated whether this practice leads to lower quality models that would perform poorly in the real clinical world. We ran a virtual clinical trial using an FDA-approved diabetes simulator to test how well models that had been trained on resampled data perform in environments with unpredictable decision intervals. We found that models trained on artificially regular decisions perform worse than expected when trialled in environments with these irregular timings. Concerningly, standard tests used to predict model safety before clinical use do not detect this, producing falsely optimistic scores. We then assessed models trained on natural, unedited data. These models performed much better and the safety tests accurately predicted their true performance. We therefore recommend training and evaluating AI models using unedited data to preserve the authentic timing of clinical practice.