Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Large Language Models able to Predict Highly Cited Papers? Evidence from Statistical Publications

Zhanshuo Ye, Yiming Hou, Rui Pan, Tianchen Gao, Hansheng Wang

TL;DR

The paper investigates whether large language models can predict long-term citation impact using only early-stage textual information from statistical papers. It introduces a structured, text-centered framework with carefully designed prompts and five-component prompts to elicit expert-like judgments about potential high impact, evaluated on a large corpus of 90,167 papers (1991–2023). Across multiple five-year windows and thresholds (Top 1%, 5%, 10%), DeepSeek-based LLMs outperform baselines and demonstrate stable generalization over time, with analyses revealing focus on topics like causal inference, inference, Bayesian methods, and ML integration. The authors also deploy a WeChat mini program to provide practical early-impact assessments, while acknowledging limitations such as prompt-dependence, potential biases, and the need to combine textual signals with other information for robust research-evaluation tools.

Abstract

Predicting highly-cited papers is a long-standing challenge due to the complex interactions of research content, scholarly communities, and temporal dynamics. Recent advances in large language models (LLMs) raise the question of whether early-stage textual information can provide useful signals of long-term scientific impact. Focusing on statistical publications, we propose a flexible, text-centered framework that leverages LLMs and structured prompt design to predict highly cited papers. Specifically, we utilize information available at the time of publication, including titles, abstracts, keywords, and limited bibliographic metadata. Using a large corpus of statistical papers, we evaluate predictive performance across multiple publication periods and alternative definitions of highly cited papers. The proposed approach achieves stable and competitive performance relative to existing methods and demonstrates strong generalization over time. Textual analysis further reveals that papers predicted as highly cited concentrate on recurring topics such as causal inference and deep learning. To facilitate practical use of the proposed approach, we further develop a WeChat mini program, \textit{Stat Highly Cited Papers}, which provides an accessible interface for early-stage citation impact assessment. Overall, our results provide empirical evidence that LLMs can capture meaningful early signals of long-term citation impact, while also highlighting their limitations as tools for research impact assessment.

Are Large Language Models able to Predict Highly Cited Papers? Evidence from Statistical Publications

TL;DR

The paper investigates whether large language models can predict long-term citation impact using only early-stage textual information from statistical papers. It introduces a structured, text-centered framework with carefully designed prompts and five-component prompts to elicit expert-like judgments about potential high impact, evaluated on a large corpus of 90,167 papers (1991–2023). Across multiple five-year windows and thresholds (Top 1%, 5%, 10%), DeepSeek-based LLMs outperform baselines and demonstrate stable generalization over time, with analyses revealing focus on topics like causal inference, inference, Bayesian methods, and ML integration. The authors also deploy a WeChat mini program to provide practical early-impact assessments, while acknowledging limitations such as prompt-dependence, potential biases, and the need to combine textual signals with other information for robust research-evaluation tools.

Abstract

Predicting highly-cited papers is a long-standing challenge due to the complex interactions of research content, scholarly communities, and temporal dynamics. Recent advances in large language models (LLMs) raise the question of whether early-stage textual information can provide useful signals of long-term scientific impact. Focusing on statistical publications, we propose a flexible, text-centered framework that leverages LLMs and structured prompt design to predict highly cited papers. Specifically, we utilize information available at the time of publication, including titles, abstracts, keywords, and limited bibliographic metadata. Using a large corpus of statistical papers, we evaluate predictive performance across multiple publication periods and alternative definitions of highly cited papers. The proposed approach achieves stable and competitive performance relative to existing methods and demonstrates strong generalization over time. Textual analysis further reveals that papers predicted as highly cited concentrate on recurring topics such as causal inference and deep learning. To facilitate practical use of the proposed approach, we further develop a WeChat mini program, \textit{Stat Highly Cited Papers}, which provides an accessible interface for early-stage citation impact assessment. Overall, our results provide empirical evidence that LLMs can capture meaningful early signals of long-term citation impact, while also highlighting their limitations as tools for research impact assessment.
Paper Structure (29 sections, 5 figures, 8 tables)

This paper contains 29 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Predictors of Citation Impact
  4. Citation Impact Prediction Models
  5. LLM-based Approaches
  6. Data Collection and Descriptive Analysis
  7. Data Collection
  8. Descriptive Analysis
  9. LLM-powered Prediction Framework
  10. Prompt Engineering
  11. LLM Interaction
  12. Results
  13. Overall Performance
  14. Comparison of LLMs
  15. The Stability of LLMs
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Left: Annual number of papers collected by our research group. The series exhibits a clear upward trend, with particularly rapid growth after 2009. Right: Log–log distribution of citation counts, showing a pronounced long-tailed pattern that approximately follows a power-law distribution.
  • Figure 2: Overall methodological framework integrating data collection, prompt design, and predictive modeling by LLM interaction.
  • Figure 3: Predicted positive rates across publication groups for ChatGPT 4o mini, Gemini 2.0 Flash, DeepSeek R1 and DeepSeek V3. The figure compares how frequently each model classifies papers as highly cited within each group.
  • Figure 4: Topic-level distribution of phrases in predicted highly cited papers. The treemap visualizes major research themes and their relative prominence based on phrase frequencies extracted from titles and abstracts.
  • Figure 5: The visual interface of the STAT Highly-Cited Papers mini program

Theorems & Definitions (1)

  • Remark 1: Potential Information Leakage