Journal article classification using abstracts: a comparison of classical and transformer-based machine learning methods

In this article we analyze the performance of existing models to classify journal articles into disciplines from a predefined classification scheme (i.e., supervised learning), based on their abstract. The first part analyzes scenarios with ample labeled data, comparing the performance of the Support Vector Machine algorithm (SVM) combined with TF-IDF and with SPECTER embeddings (Cohan et al. SPECTER: Document-level representation learning using citation-informed transformers, https://doi.org/10.48550/arXiv.2004.07180 , 2020) and Bidirectional Encoder Representations from Transformers (BERT) models. The second part employes Generative Pre-trained Transformer model 3.5 turbo (GPT-3.5-turbo) for the zero- and few-shot learning situations. Through the use of GPT-3.5-turbo we examine how different characterizations of disciplines (such as names, descriptions, and examples) affect the model’s ability to classify articles. The data set comprises journal articles published in 2022 and indexed in the Web of Science, with subject categories aligned to a modified version of the OECD Fields of Research and Development (FoRD) classification scheme. We find that BERT models surpass the SVM + TF-IDF baseline and SVM + SPECTER in all areas. For all disciplinary areas except Humanities, we observe minimal variation among the models fine-tuned on larger datasets, and greater variability with smaller training datasets. The GPT 3.5-turbo results show significant fluctuations across disciplines, influenced by the clarity of their definition and their distinctiveness as research topics compared to other fields. Although the two approaches are not directly comparable, we conclude that the classification models show promising results in their specific scenarios, with variations across disciplines.