LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark

TL;DR

LongBench Pro addresses the need for realistic, scalable long-context evaluation by providing a bilingual English–Chinese benchmark with 1,500 samples across 11 primary tasks and 25 secondary tasks, spanning 8k–256k tokens. It introduces a multi-dimensional taxonomy for context requirement, length, and difficulty and a Human-Model Collaborative Construction pipeline to balance authenticity with efficiency. Evaluating 46 long-context LLMs reveals that context-length optimization often surpasses parameter scaling, that effective context length is typically shorter than claimed with cross-lingual gaps, and that thinking-enabled or mixed-thinking prompting yields substantial gains, especially when native reasoning is trained. Overall, LongBench Pro offers a robust, scalable testbed for advancing long-context understanding and cross-lingual robustness in next-generation systems.

Abstract

The rapid expansion of context length in large language models (LLMs) has outpaced existing evaluation benchmarks. Current long-context benchmarks often trade off scalability and realism: synthetic tasks underrepresent real-world complexity, while fully manual annotation is costly to scale to extreme lengths and diverse scenarios. We present LongBench Pro, a more realistic and comprehensive bilingual benchmark of 1,500 naturally occurring long-context samples in English and Chinese spanning 11 primary tasks and 25 secondary tasks, with input lengths from 8k to 256k tokens. LongBench Pro supports fine-grained analysis with task-specific metrics and a multi-dimensional taxonomy of context requirement (full vs. partial dependency), length (six levels), and difficulty (four levels calibrated by model performance). To balance quality with scalability, we propose a Human-Model Collaborative Construction pipeline: frontier LLMs draft challenging questions and reference answers, along with design rationales and solution processes, to reduce the cost of expert verification. Experts then rigorously validate correctness and refine problematic cases. Evaluating 46 widely used long-context LLMs on LongBench Pro yields three findings: (1) long-context optimization contributes more to long-context comprehension than parameter scaling; (2) effective context length is typically shorter than the claimed context length, with pronounced cross-lingual misalignment; and (3) the "thinking" paradigm helps primarily models trained with native reasoning, while mixed-thinking designs offer a promising Pareto trade-off. In summary, LongBench Pro provides a robust testbed for advancing long-context understanding.

Figures (11)

• Figure 1: Performance of advanced long-context LLMs on LongBench Pro.
• Figure 2: Task mapping between LongBench Pro and existing benchmarks.
• Figure 3: The construction process of LongBench Pro includes document collection, human–model collaborative sample generation, question standardization, answer review, and difficulty classification.
• Figure 4: Overview of LongBench Pro sample distributions.
• Figure 5: Trends in Best-of-N metrics.