Predicting citation impact of research papers using GPT and other text embeddings

The impact of research papers, typically measured in terms of citation counts, depends on several factors, including the reputation of the authors, journals, and institutions, in addition to the quality of the scientific work. In this paper, we present an approach that combines natural language processing and machine learning to predict the impact of papers in a specific journal. Our focus is on the text, which should correlate with impact and the topics covered in the research. We employed a dataset of over 40,000 articles from ACS Applied Materials and Interfaces spanning from 2012 to 2022. The data was processed using various text embedding techniques and classified with supervised machine learning algorithms. Papers were categorized into the top 20% most cited within the journal, using both yearly and cumulative citation counts as metrics. Our analysis reveals that the method employing an embedding model based on generative pre-trained transformers (GPT) was the most efficient for embedding, while the random forest algorithm exhibited the best predictive power among the machine learning algorithms. An optimized accuracy of 80% in predicting whether a paper was among the top 20% most cited was achieved for the cumulative citation count when abstracts were processed. This accuracy is noteworthy, especially considering that information about authors, institutions, and early citation pattern was not taken into account. The accuracy increased only slightly when the full texts of the papers were processed. Also significant is the finding that the term frequency–inverse document frequency (TFIDF) embedding method, despite its simplicity, achieved a performance comparable to that of GPT. Since TFIDF captures the topics of the paper, we infer that, apart from possible biases related to authors and institution, citation counts for the journal examined may be predicted by identifying topics and “reading” the abstract of a paper.