Portability of a text mining algorithm for detecting adverse drug reactions in electronic health records across diverse patient groups in two Dutch hospitals

Adverse Drug Reactions (ADRs) pose a significant challenge in healthcare. While structured documentation of ADRs in electronic health records (EHRs) enables automated alerting, many ADRs are recorded as unstructured free-text, limiting detection. Text mining (TM) shows potential for extracting clinically relevant data from unstructured text. However, the portability of TM algorithms across different institutions and departments remains uncertain, due to variations in EHR structures and documentation practices. To enhance these general-purpose algorithms, evaluating their portability is essential for ensuring effective performance across diverse clinical settings. To evaluate the portability of a previously developed TM-based ADR identification algorithm by assessing its performance using EHRs from two different departments in two different hospitals. EHR free-text data from 62 hospitalized patients in the geriatric and orthopedic departments of two Dutch teaching hospitals were reviewed for ADRs via manual review and the TM algorithm. Performance was evaluated using F-score, sensitivity and positive predictive value (PPV), with comparisons across hospitals and departments. Manual review identified 359 unique ADRs. The TM algorithm detected 534 potential ADRs (pADRs), 286 of which overlapped with manual review, yielding an F-score of 0.64, sensitivity of 80% and PPV of 54%. Performance was consistent across hospitals and departments. Notably, 26 pADRs identified by the algorithm were clinically relevant yet missed in manual review. This study demonstrates portability of the TM algorithm by identifying pADRs across different hospitals and departments without adaptations. These findings support its broader implementation potential for ADR detection in diverse healthcare settings.Author summary: Adverse Drug Reactions (ADRs) present a significant challenge in healthcare, with many being recorded as unstructured free-text in Electronic Health Records (EHRs). This study evaluates the portability of a text mining (TM) algorithm developed for ADR identification in EHRs, by assessing its performance across two departments in two Dutch hospitals. EHR free-text data from 62 patients in the geriatric and orthopedic departments were analyzed using manual review and the TM algorithm. The results showed that the TM algorithm demonstrated a good performance, with an F-score of 0.64, sensitivity of 80%, and positive predictive value (PPV) of 54%. Additionally, the algorithm identified clinically relevant ADRs that were missed in manual review. These findings suggest that the TM algorithm is portable across different clinical settings without requiring adaptation, highlighting its potential for broader implementation in ADR detection.