ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations

TL;DR

ScholarCopilot addresses hallucinated citations in long-form academic writing by unifying text generation with context-aware retrieval through dynamic retrieval tokens that interleave citation queries with writing. It trains a single model (building on Qwen-2.5-7B) with a joint objective that couples next-token generation $L_g$ and a contrastive retrieval loss $L_r$, enabling seamless content augmentation from a large arXiv CS corpus. Empirically, it achieves a top-1 retrieval recall of $40.1\%$ and a generation quality of $16.21/25$, outperforming larger baselines and even rivaling ground-truth citation setups; extensive human studies report strong citation quality advantages over ChatGPT. The work demonstrates that tightly integrating generation and retrieval in an iterative, human-in-the-loop framework yields more accurate, coherent, and citation-aware academic writing, with potential to improve bibliographic integrity in AI-assisted scholarly workflows.

Abstract

Academic writing requires both coherent text generation and precise citation of relevant literature. Although recent Retrieval-Augmented Generation (RAG) systems have significantly improved factual accuracy in general-purpose text generation, their ability to support professional academic writing remains limited. In this work, we introduce ScholarCopilot, a unified framework designed to enhance existing large language models for generating professional academic articles with accurate and contextually relevant citations. ScholarCopilot dynamically determines when to retrieve scholarly references by generating a retrieval token [RET], which is then used to query a citation database. The retrieved references are fed into the model to augment the generation process. We jointly optimize both the generation and citation tasks within a single framework to improve efficiency. Our model is built upon Qwen-2.5-7B and trained on 500K papers from arXiv. It achieves a top-1 retrieval accuracy of 40.1% on our evaluation dataset, outperforming baselines such as E5-Mistral-7B-Instruct (15.0%) and BM25 (9.8%). On a dataset of 1,000 academic writing samples, ScholarCopilot scores 16.2/25 in generation quality -- measured across relevance, coherence, academic rigor, completeness, and innovation -- significantly surpassing all existing models, including much larger ones like the Retrieval-Augmented Qwen2.5-72B-Instruct. Human studies further demonstrate that ScholarCopilot, despite being a 7B model, significantly outperforms ChatGPT, achieving 100% preference in citation quality and over 70% in overall usefulness.

Figures (10)

• Figure 1: Comparison of traditional Retrieval-Augmented Generation (RAG) systems and our proposed ScholarCopilot. Traditional RAG systems (left) separately perform retrieval and generation, leading to representation misalignment. In contrast, ScholarCopilot (right) dynamically generates retrieval tokens ([RET]) during text generation for integrated and context-aware reference retrieval.
• Figure 2: Comparison between traditional Retrieval-Augmented Generation (RAG) methods (left) and ScholarCopilot (right). Traditional RAG follows a static retrieval-then-generation pipeline, retrieving references independently before generation. ScholarCopilot dynamically interleaves retrieval and generation by producing retrieval tokens ([RET]) based on current context, enabling context-aware citation retrieval and optional user refinement.
• Figure 3: The pipeline for creating the ScholarCopilot dataset. Our final dataset includes 10M citations matched from arXiv and 6.8M citations matched from Semantic Scholar (one paper may be cited by multiple articles). However, at inference time, to ensure reference quality, we only use the 670K articles from arXiv as the corpus.
• Figure 4: Unified training framework of ScholarCopilot. The architecture jointly optimizes the next token prediction loss for text generation and the contrastive loss for citation retrieval. Retrieval tokens ([RET]) dynamically trigger retrieval. $<q, d^+>$ indicates the positive pair of query and document during contrastive learning, and $<q, d^->$ indicates the negative pair. The generation model and retrieval model share parameters. In this figure, Paper 1 and Paper 2 can be considered as hard negatives for each other.
• Figure 5: Comparison of citation retrieval performance (Recall@k) between ScholarCopilot and baseline retrieval methods (BM25 and E5-Mistral-7B-Instruct).