ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers

TL;DR

ForeCite introduces an end-to-end regression framework that adapts pre-trained causal transformers to predict future average monthly citation rates from manuscript text. By training a simple linear head atop a causal LM and employing techniques like QLoRA finetuning and 4-bit NF4 quantization, it achieves a leading $ρ=0.826$ on 900K biomedical papers and demonstrates consistent scaling gains across model sizes and data volumes. Gradient-based saliency reveals a strong emphasis on titles and abstracts, suggesting surrogate signals rather than solely deep textual content drive predictions, while temporal holdout indicates drift that motivates online updating. Overall, the work sets a new state-of-the-art in citation forecasting and outlines practical, scalable paths for AI-assisted research evaluation, including multi-modal extensions and online-learning strategies.

Abstract

Predicting the future citation rates of academic papers is an important step toward the automation of research evaluation and the acceleration of scientific progress. We present $\textbf{ForeCite}$, a simple but powerful framework to append pre-trained causal language models with a linear head for average monthly citation rate prediction. Adapting transformers for regression tasks, ForeCite achieves a test correlation of $ρ= 0.826$ on a curated dataset of 900K+ biomedical papers published between 2000 and 2024, a 27-point improvement over the previous state-of-the-art. Comprehensive scaling-law analysis reveals consistent gains across model sizes and data volumes, while temporal holdout experiments confirm practical robustness. Gradient-based saliency heatmaps suggest a potentially undue reliance on titles and abstract texts. These results establish a new state-of-the-art in forecasting the long-term influence of academic research and lay the groundwork for the automated, high-fidelity evaluation of scientific contributions.

Figures (12)

• Figure 1: Distribution of articles in the final dataset containing each domain-specific keyword.
• Figure 2: Standardized average monthly citation rates post log-transform, overlaid with a scaled standard normal curve for reference.
• Figure 3: A 3D visualization of the relationship between model size (in billions of parameters, $\log_2$ scaled), data volume (in percent of total corpus, $\log_2$ scaled), and performance (in Pearson correlation, evaluated on a holdout test set.)
• Figure 4: This graph visualizes the rolling test correlation of the Bloom-560m model, evaluated with temporal holdout. For example, the point with $x$ value 2024-10 and $y$ value 0.530 represents a 0.530 Pearson correlation between the true and predicted target values between 2023-01 and 2024-10. The blue line denotes the original $r$ found during the scaling-law evaluation of Bloom-560m.
• Figure 5: A table of hyperparameters used for all experiments. Any omitted parameters were left at their default values.