LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion

TL;DR

LongReason addresses a gap in evaluating long-context reasoning by introducing a synthetic, controllable benchmark that expands short-context questions into long-context variants. The authors present a context-expansion pipeline powered by LLMs and self-verification to generate 794 MCQs across reading comprehension, logical inference, and mathematical word problems, with context lengths adjustable up to 128K tokens. Evaluating 21 LLMs shows significant performance drops as context length increases, with closed-source models like Gemini-1.5 Pro handling long contexts better than many open-source counterparts, yet still leaving ample room for improvement. The dataset is open-sourced to support rigorous benchmarking and future research in robust long-context reasoning.

Abstract

Large language models (LLMs) have demonstrated remarkable progress in understanding long-context inputs. However, benchmarks for evaluating the long-context reasoning abilities of LLMs fall behind the pace. Existing benchmarks often focus on a narrow range of tasks or those that do not demand complex reasoning. To address this gap and enable a more comprehensive evaluation of the long-context reasoning capabilities of current LLMs, we propose a new synthetic benchmark, LongReason, which is constructed by synthesizing long-context reasoning questions from a varied set of short-context reasoning questions through context expansion. LongReason consists of 794 multiple-choice reasoning questions with diverse reasoning patterns across three task categories: reading comprehension, logical inference, and mathematical word problems. We evaluate 21 LLMs on LongReason, revealing that most models experience significant performance drops as context length increases. Our further analysis shows that even state-of-the-art LLMs still have significant room for improvement in providing robust reasoning across different tasks. We have open-sourced LongReason under https://huggingface.co/datasets/lz1bytedance/LongReason to support the comprehensive evaluation of LLMs' long-context reasoning capabilities.

Figures (6)

• Figure 1: Overview of our pipeline for constructing LongReason. Givem a short reasoning question $Q_{\text{short}}$, the pipeline first separates it into a background context $C_{\text{short}}$ and a final question $I$. Next, multiple paragraphs are synthesized from the background context $C_{\text{short}}$. These synthesized paragraphs are then embedded within irrelevant passages to create a long-context background. Finally, the constructed context is combined with the final question to generate the long-context reasoning question $Q_{\text{long}}$.
• Figure 2: An illustrative example in LongReason. The original question is first decomposed into a separate background passage and an inquiry based on it. The inquiry includes keywords such as “Jack’s father’s age” and a time reference like “on a sunny afternoon” from the background passage, ensuring a clear connection to the passage. Subsequently, the background passage is expanded into multiple independent materials while preserving these key keywords. Finally, these independent materials are combined with some unrelated passages to create the final long-context reasoning question.
• Figure 3: The number of reasoning steps in the ground-truth analysis for questions in LongReason.
• Figure 4: Performance of the Qwen2.5 series on LongReason, with model sizes ranging from 7B to 72B.
• Figure 5: Comparison of the long-context reasoning performance between Gemini-1.5 Pro and Claude 3.5-Sonnet across different task categories. In the figure, the dotted line represents the single-hop version of the synthesized questions, where all clues are placed together in the context. The solid line represents the multi-hop version, which is the standard format used in LongReason, where clues are distributed separately throughout the context.