Doctors vs. Algorithms: Physicians, too, struggle to learn from evidence that contradicts AI suggestions

Despite their widespread adoption, Artificial Intelligence-based Patient Classification Systems sometimes rely on incorrect, outdated, or incomplete data, which can lead to inaccurate outputs. Nevertheless, health professionals are expected to override these errors, at least when they have access to critical information. To test this, we conducted two experiments in which professional physicians interacted with an Artificial Intelligence system that incorrectly classified fictitious patients as either highly or lowly sensitive to a treatment. The physicians administered the treatment to a series of fictitious patients and received feedback that was useful for learning that the patient classification was incorrect and that all patients were equally sensitive to the treatment. We ran two experiments: in Experiment 1, the medicine showed medium effectiveness for both types of patients, while in Experiment 2, the treatment was completely ineffective for both types of patients. The results showed that, in the two experiments, physicians generally trusted the AI-based patient classification and struggled to learn from the evidence. Furthermore, in Experiment 2, they failed to realize that the treatment was ineffective. Our findings have important implications for healthcare professionals and patients, underscoring the need to critically evaluate Patient Classification Systems.Author summary: Artificial Intelligence (AI) is now widely used in various fields, including clinical practice, where AI systems assist in classifying patients for diagnosis and treatment. However, these AI-based patient classification systems can sometimes use incorrect, outdated, or incomplete data, leading to errors. In such cases, health professionals are expected to catch and correct these errors. Earlier studies found that the general population often do not sufficiently detect and fix AI’s mistakes. We wanted to see if professional physicians would act differently so that they would notice and correct these errors. To find out, we conducted two experiments in which participants had to decide whether to administer a fictitious treatment to patients who were incorrectly classified by an AI as being more or less sensitive. In the first experiment, the treatment worked moderately (and equally) well for both groups. In the second, the treatment did not work at all for either group. In both experiments, physicians trusted the AI’s classifications and did not use the available information to override the erroneous classification. What is more, in the second experiment, they did not even realize that the treatment was completely ineffective.