AcademicEval: Live Long-Context LLM Benchmark

TL;DR

AcademicEval tackles the core challenge of evaluating LLMs on ultra-long, hierarchically structured academic writing tasks in a live, leakage-resistant setting by leveraging arXiv papers and a co-author graph for context-sensitive demonstrations.The benchmark offers four tasks (Title, Abstract, Introduction, Related Work) with flexible input lengths and periodic live updates to prevent data leakage, enabling both automatic metric and LLM-based qualitative evaluation of long-context modeling capabilities.Key findings show automatic metrics often favor retrieval-augmented approaches and degrade with longer contexts, while LLM-as-a-Judge reveals task-dependent preferences and highlights gaps in current long-context understanding, suggesting complementary roles for automatic and judge-based assessments.Together, these results provide practical insights into improving long-context modeling, such as leveraging graph-informed demonstrations, addressing leakage in live benchmarks, and guiding future benchmark design toward flexible context management and holistic quality evaluation.

Abstract

Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose \textsc{AcademicEval}, a live benchmark for evaluating LLMs over long-context generation tasks. \textsc{AcademicEval} adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, \textit{i.e.}, \textsc{Title}, \textsc{Abstract}, \textsc{Introduction}, and \textsc{Related Work}, which cover a wide range of abstraction levels and require no manual labeling. Moreover, \textsc{AcademicEval} integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, \textsc{AcademicEval} features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on \textsc{AcademicEval}, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs' long-context modeling capabilities. Code is available at https://github.com/ulab-uiuc/AcademicEval

Figures (3)

• Figure 1: AcademicEval Benchmark. We construct a co-author graph via arXiv and conduct a chronological split on all paper samples (training, validation, and test samples are represented by red, orange, and green, respectively). Each paper sample is preprocessed into separate sections and can be integrated with few-shot demonstrations from co-author papers.
• Figure 2: Live Evaluation of AcademicEval Benchmark. To support continual benchmarking, AcademicEval incrementally updates the co-author graph using daily arXiv data. The procedure includes: (1) Node Update – augmenting node features for authors with newly published first-author papers; (2) Node and Edge Update – identifying and prioritizing new co-authors via BFS to expand the graph with recent publications; and (3) Graph Pruning – removing outdated papers and inactive authors to maintain graph connectivity and efficiency.
• Figure 3: Analysis of RALM on Abs-9K. The left figure shows results with Gemma Instruct (7B), while the right one shows results with LLaMA-3 Chat (70B).