What Should I Cite? A RAG Benchmark for Academic Citation Prediction

TL;DR

CiteRAG tackles the problem of navigating the rapidly expanding corpus of scientific literature by introducing a comprehensive RAG-based benchmark for academic citation prediction. It defines two granular tasks (list-level and position-level), builds a large three-level hierarchical corpus, and proposes a multi-level hybrid retrieval architecture coupled with contrastive fine-tuning to ground generation in real citations. The study presents extensive experiments across closed- and open-source LLMs, showing that retrieval augmentation and domain-specific fine-tuning yield substantial improvements and reduce hallucinations relative to baseline methods. An open-source toolkit accompanies the benchmark, enabling reproducible evaluation and serving as a methodological template for citation prediction and potentially other scientific domains.

Abstract

With the rapid growth of Web-based academic publications, more and more papers are being published annually, making it increasingly difficult to find relevant prior work. Citation prediction aims to automatically suggest appropriate references, helping scholars navigate the expanding scientific literature. Here we present \textbf{CiteRAG}, the first comprehensive retrieval-augmented generation (RAG)-integrated benchmark for evaluating large language models on academic citation prediction, featuring a multi-level retrieval strategy, specialized retrievers, and generators. Our benchmark makes four core contributions: (1) We establish two instances of the citation prediction task with different granularity. Task 1 focuses on coarse-grained list-specific citation prediction, while Task 2 targets fine-grained position-specific citation prediction. To enhance these two tasks, we build a dataset containing 7,267 instances for Task 1 and 8,541 instances for Task 2, enabling comprehensive evaluation of both retrieval and generation. (2) We construct a three-level large-scale corpus with 554k papers spanning many major subfields, using an incremental pipeline. (3) We propose a multi-level hybrid RAG approach for citation prediction, fine-tuning embedding models with contrastive learning to capture complex citation relationships, paired with specialized generation models. (4) We conduct extensive experiments across state-of-the-art language models, including closed-source APIs, open-source models, and our fine-tuned generators, demonstrating the effectiveness of our framework. Our open-source toolkit enables reproducible evaluation and focuses on academic literature, providing the first comprehensive evaluation framework for citation prediction and serving as a methodological template for other scientific domains. Our source code and data are released at https://github.com/LQgdwind/CiteRAG.

Figures (8)

• Figure 1: Overview of the CiteRAG benchmark pipeline comprising three stages: (1) Data Collection and Corpus Pre-processing constructs a three-level hierarchical corpus from web-crawled papers; (2) Dataset Pre-processing filters and formats data into Task 1 and Task 2 QA pairs with test instances removed from the corpus; (3) Retrieval-Augmented Generation Pipeline trains embedding models and retrievers, then applies multi-level hybrid retrieval to feed generators for task-specific citation prediction and evaluation.
• Figure 2: Corpus and dataset statistics. The left sunburst chart shows three-level organization with 554,719 total papers across 10 fields, training/test distributions, and metadata structure. The upper right panels display citation frequency distributions for Task 1 and Task 2. The lower right panels show token composition breakdowns and response token length distributions for both tasks.
• Figure 3: Performance Comparison of Single-Level vs. Multi-Level Fused Retrieval Strategies
• Figure 4: Question-answer format specifications for Task 1 and Task 2.
• Figure 5: Citation Quality Assessment: Diversity Entropy and Hallucination Rate Across Models.