From Words to Worth: Newborn Article Impact Prediction with LLM

TL;DR

This study tackles newborn article impact prediction by predicting a time-normalized, cross-field impact metric, TNCSI$_{\mathrm{SP}}$, from only titles and abstracts. It introduces two datasets (TKPD and NAID) and demonstrates that fine-tuned LLMs (notably LLaMA-3) can regress $\hat{\mathrm{TNCSI}}_{\mathrm{SP}}$ for newborn papers with state-of-the-art accuracy (MAE $0.216$, NDCG@20 $0.901$) while using resource-efficient techniques (LoRA and 8-bit quantization). The approach emphasizes semantic feature extraction via prompts and a first-token regression head, achieving robust performance without external data and showing real-world applicability for journal-level impact estimation and arXiv triage. Overall, the work shifts focus from external bibliometrics to content-driven impact prediction and delivers practical tools and datasets for automated scholarly screening.

Abstract

As the academic landscape expands, the challenge of efficiently identifying impactful newly published articles grows increasingly vital. This paper introduces a promising approach, leveraging the capabilities of LLMs to predict the future impact of newborn articles solely based on titles and abstracts. Moving beyond traditional methods heavily reliant on external information, the proposed method employs LLM to discern the shared semantic features of highly impactful papers from a large collection of title-abstract pairs. These semantic features are further utilized to predict the proposed indicator, TNCSI_SP, which incorporates favorable normalization properties across value, field, and time. To facilitate parameter-efficient fine-tuning of the LLM, we have also meticulously curated a dataset containing over 12,000 entries, each annotated with titles, abstracts, and their corresponding TNCSI_SP values. The quantitative results, with an MAE of 0.216 and an NDCG@20 of 0.901, demonstrate that the proposed approach achieves state-of-the-art performance in predicting the impact of newborn articles when compared to several promising methods. Finally, we present a real-world application example for predicting the impact of newborn journal articles to demonstrate its noteworthy practical value. Overall, our findings challenge existing paradigms and propose a shift towards a more content-focused prediction of academic impact, offering new insights for article impact prediction.

Figures (6)

• Figure 1: A taxonomy of article impact prediction (AIP): since there are virtually no other Lv. II methods, the "newborn AIP" segment represents the proposed approach, which predicts future academic impact in a "double-blind peer-review" manner.
• Figure 2: Flowchart for calculating $\mathrm{{TNCSI}_{SP}}$: ${\mathrm{TNCSI}}_{\mathrm{SP}} \in [0, 1]$ represents the probability that a paper's citation count outperforms other papers in the same field and time period. "S2" refers to Semantic Scholar.
• Figure 3: LLM as scholar impact predictor: overall framework of the proposed approach. Only the Next Token (first generated token) is used to regress the ${\mathrm{TNCSI}}_{\mathrm{SP}}$.
• Figure 4: The impact of various model parameters on performance: the larger the number of model parameters, the better the performance.
• Figure 5: Impact of different prediction targets on performance: $\mathrm{{TNCSI}_{SP}}$ demonstrates superior performance over ${\mathrm{TNCSI}}$ with training data from different years.