How to use assignments of United Nations sustainable development goals (SDGs) to scientific papers in research evaluation? The proposal of a gold standard combining assignments from different data providers

To identify research that addresses the biggest problems facing the world today, researchers have used the United Nations (UN) Sustainable Development Goals (SDGs) as a blueprint for measuring the societal impact of research. Although one might assume that the mapping of papers to the SDGs has a high degree of uniformity (based more or less on the same data), empirical results show that this is not the case: different mapping approaches lead to different assignments. The central aim of this paper is to use the National Open Research Analytics (NORA, Technical University of Denmark) database not only to test the agreement between four SDG mapping approaches [Web of Science (WoS)TM, OSDG, Scopus-ML, Scopus-SM], but also to investigate whether an overarching SDG mapping can be found (a gold standard). N = 526,520 Danish publications are analyzed with respect to the following aspects: distribution of SDGs over publications, agreement between SDG mapping approaches, prediction of SDG assignments, and the gold standard combining assignments from different mapping approaches (occupancy analysis). The main findings are as follows: 35.5% of the documents have at least one SDG assignment. The SDG occupancy is low, with the highest occupancy and detection probability for SDG 3 “Good health and well-being”. Agreement between SDG mapping approaches is low to moderate. The result of the gold standard analysis is not only the determination of a “true” SDG status of a publication and the frequency of the SDG assignments (occupancy), but also the uncertainty associated with this SDG assignment (detection probability).