Predicting citation impact of research papers using GPT and other text embeddings

TL;DR

This study investigates whether paper content alone can predict a journal's citation impact, focusing on ACS AMI papers from 2012–2022. It compares GPT-based embeddings and traditional text representations across abstracts and full text, using ACC and YCC as targets and evaluating with AUC-ROC and AUC-PR on balanced and skewed datasets. The main finding is that GPT embeddings paired with a Random Forest classifier achieve the strongest, roughly 0.80 accuracy on ACC, with TFIDF providing a strong baseline and abstract-only text often matching or exceeding full-text performance in skewed data. The results suggest that topic signals and writing style captured by abstracts largely explain citation potential, offering practical implications for editors and researchers while highlighting ethical considerations around bias and interpretability. Future work includes integrating author-level signals, combining LLMs with graph-based analyses, and assessing bias and interpretability of content-based citation predictions.

Abstract

The impact of research papers, typically measured in terms of citation counts, depends on several factors, including the reputation of the authors, journals, and institutions, in addition to the quality of the scientific work. In this paper, we present an approach that combines natural language processing and machine learning to predict the impact of papers in a specific journal. Our focus is on the text, which should correlate with impact and the topics covered in the research. We employed a dataset of over 40,000 articles from ACS Applied Materials and Interfaces spanning from 2012 to 2022. The data was processed using various text embedding techniques and classified with supervised machine learning algorithms. Papers were categorized into the top 20% most cited within the journal, using both yearly and cumulative citation counts as metrics. Our analysis reveals that the method employing generative pre-trained transformers (GPT) was the most efficient for embedding, while the random forest algorithm exhibited the best predictive power among the machine learning algorithms. An optimized accuracy of 80\% in predicting whether a paper was among the top 20% most cited was achieved for the cumulative citation count when abstracts were processed. This accuracy is noteworthy, considering that author, institution, and early citation pattern information were not taken into account. The accuracy increased only slightly when the full texts of the papers were processed. Also significant is the finding that a simpler embedding technique, term frequency-inverse document frequency (TFIDF), yielded performance close to that of GPT. Since TFIDF captures the topics of the paper we infer that, apart from considering author and institution biases, citation counts for the considered journal may be predicted by identifying topics and "reading" the abstract of a paper.

Figures (7)

• Figure 1: Comparative performance of text embedding models on citation prediction. Box plots illustrating the effectiveness of different embedding techniques in predicting the top 20% cited academic papers over the years using different classification algorithms. Performance is evaluated using (AUC-ROC) and (AUC-PR) across the balanced and skewed datasets. ACC denotes the prediction of papers in the top 20% of accumulative citations over the years, while YCC refers to the prediction of papers with the top annual citations in a specific year. The results highlight the variance in model performance, with implications for model selection in citation analysis tasks.
• Figure 2: Comparative performance of machine learning classification algorithms on citation prediction. The box plots illustrate the effectiveness of the different embedding techniques in predicting the top 20% cited academic papers over the years using different classification algorithms. Performance is evaluated using the Area Under the ROC Curve and of the Precision-Recall Curve (AUC-PR) for the balanced and skewed datasets. ACC denotes the prediction of papers in the top 20% of accumulative citations over the years, while YCC refers to the prediction of papers with the top annual citations in a specific year. The results highlight the variance in model performance, with implications for model selection in citation analysis tasks.
• Figure 3: Yearly performance trends of text embedding models for predicting top 20% accumulative citations in academic papers over an 11-Year horizon using AUC-ROC and AUC-PR metrics on balanced and skewed datasets. Here we used the ACC metric as a measure of performance.
• Figure 4: Yearly performance trends of text embedding models for predicting top 20% annual citations in academic papers over an 11-Year Horizon using AUC-ROC and AUC-PR metrics on balanced and skewed datasets. Here we used the YCC metric as a measure of performance.
• Figure 5: Comparative analysis of random forest with 1000 trees and GPT embeddings for Full-Text versus abstract-only for academic papers performance over 11 years post-publication using AUC and AUC PR performance metrics across balanced and skewed datasets.