How similar are field-normalized citation impact scores obtained from OpenAlex and three popular commercial databases? An empirical comparison based on large German universities

OpenAlex is a freely available bibliographic database that can be used for bibliometric studies. In this study, we compared certain field-normalized citation scores (NCS) from OpenAlex with those from three commercial databases (Web of Science, Scopus, and Dimensions). We were interested in the question whether the NCS from OpenAlex are comparable to those from the commercial databases and can be alternatively used in evaluative bibliometrics. The NCS have been calculated for nearly 335,000 papers published by 48 German universities in four main subject areas between 2013 and 2017. We found varying but overall strong agreement between the scores according to Lin’s concordance correlation coefficient. Separating the publication set along the single universities and moreover along the four main subject areas involved revealed significant differences at the level of single papers but also gave indications on how to possibly mitigate outlier cases. We calculated mean normalized citation scores for the 48 universities and found that the agreements across different databases are low. On the one hand, the results suggest that comparisons of universities using NCS across different databases should be avoided. On the other hand, the difference of the concordance correlation coefficients at paper and university level is a good example for the problem of ecological fallacy in bibliometrics: The mean impact is not representative for the single papers’ impact in the set.