From evidence to impact: The use of scientific research in policy documents

We examine the use of scientific research in the development of policy documents within the context of clinical practice guidelines (CPGs) for diagnosing, treating, and managing diabetes. Using natural language processing, we identify “hidden citations” (i.e., textual credit without formal citations) and “token citations” (i.e., formal citations without textual credit) to scientific research within CPGs to understand how scientific evidence is selected and integrated. We find that both types of citations are pervasive, calling into question the use of formal citations alone in understanding the societal impact of scientific research. Using data on scholarly citations and expert ratings, we find that hidden citations are positively associated with the actual impact of the research on patients and caregivers while token citations associate positively with scientific impact. Qualitative insights gathered from interviews with senior guideline writers further illustrate the reasons for certain functions of scientific research, which involve balancing scientific rigor with practical demands in the guideline writing process, the need for local adaptations, political dynamics on the organizational level, and individual preferences towards certain types of studies or the use of experiential knowledge. Our work underscores the critical role of research utilization in translating scientific evidence into policy, showing that policymaker decisions shape societal impact as much as the engagement efforts of scientists, and extends institutional accounts of symbolic and substantive knowledge use.