Does LLM Assistance Improve Healthcare Delivery ? An Evaluation Using On-Site Physicians and Laboratory Tests

This study tests the effects of large language model (LLM) decision support on patient care at two outpatient clinics in Nigeria. Health workers were given the option to make revisions to their initial care plan based on LLM feedback. The unassisted and assisted plans are evaluated using (1) comparisons with independent care plans created by on-site physicians, (2) laboratory tests for malaria, anemia, and urinary tract infections, and (3) a blinded randomized assessment by the on-site physician who saw the same patient. In response to LLM feedback, health workers changed their prescribing for more than half of the patients and reported high satisfaction with the recommendations. In a selected sample, retrospective review by academic physicians also suggested improvements in care related to long-term risk management. However, the three metrics show mixed effects of LLM-assistance, with on average no significant improvement in diagnostic alignment with physicians, detection rates for the tested conditions, or physician subjective assessments. Health workers follow LLM recommendations that agree with the physician's decisions only slightly more often than those that do not. These results suggest that, despite some benefits, LLM-based frontline health worker support is not yet a public health priority in low- and middle-income countries.