Are Nature Index journals a valid basis for academic assessment: a study of academic impact and disruptive innovation assessment based on open bibliographic metadata and citation data

Journal listings play an important role in current journal evaluation, publication reference, research assessment and position promotion. Considering the current orientation of science and technology development and the need of transformative changes in academic evaluation, this study aims to examine Nature Index journals list. The focus will be on exploring the coverage of research topics, assessing the innovation levels of the published papers, and identifying underrepresented research areas. The outcome will facilitate the identification of high-level science and technology journals that can serve as valuable references for the Nature Index team and relevant management and research institutions in their scientific research evaluation processes. The results of the study show that of the 3029 research topics in the science fields defined by OpenAlex, 2217 (73.19%) are covered by Nature Index journals. Across all 18 research disciplines outside of Dentistry and Veterinary, Nature Index journals show a clear advantage in academic quality (the average share of dominant research topics in each disciplinary category is around 90%). However, Nature Index journals have an advantage in terms of academic impact over the level of disruptive innovation. In this study, two journals have been also selected to supplement the latest Nature Index list by taking the disruptive innovation level as the core index and the academic influence and long-term development of the journals in the related research fields as the reference. Generally speaking, the list of Nature Index journals is relatively complete, but there is still room for adjustment. Nature Index journals have a high academic impact and disruptive innovation level in most of the research topics that they cover, but they are more focused on academic impact. Appropriately adding representative journals in uncharted research areas will enhance the application value of the Nature Index in academic evaluation.