InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

TL;DR

InftyThink+ addresses the fundamental bottlenecks of long-horizon reasoning in large language models by decoupling reasoning depth from context length through iterative, summary-driven reasoning. It introduces an end-to-end reinforcement learning framework with a cold-start supervised phase and trajectory-level RL, optimizing when to compress, what to preserve, and how to continue across iterations, guided by task and efficiency rewards. Empirical results on diverse benchmarks show substantial accuracy gains (e.g., +$21$ points on AIME24) and large inference-time efficiency improvements, with stronger out-of-distribution generalization and faster RL training. The work demonstrates that trajectory-level optimization, adaptive timing, and efficiency-aware objectives yield a practical, scalable pathway toward effective infinite-horizon reasoning in real-world settings.

Abstract

Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects. Iterative reasoning mitigates these issues by periodically summarizing intermediate thoughts, yet existing methods rely on supervised learning or fixed heuristics and fail to optimize when to summarize, what to preserve, and how to resume reasoning. We propose InftyThink+, an end-to-end reinforcement learning framework that optimizes the entire iterative reasoning trajectory, building on model-controlled iteration boundaries and explicit summarization. InftyThink+ adopts a two-stage training scheme with supervised cold-start followed by trajectory-level reinforcement learning, enabling the model to learn strategic summarization and continuation decisions. Experiments on DeepSeek-R1-Distill-Qwen-1.5B show that InftyThink+ improves accuracy by 21% on AIME24 and outperforms conventional long chain-of-thought reinforcement learning by a clear margin, while also generalizing better to out-of-distribution benchmarks. Moreover, InftyThink+ significantly reduces inference latency and accelerates reinforcement learning training, demonstrating improved reasoning efficiency alongside stronger performance.

Figures (29)

• Figure 1: InftyThink reasoning paradigm VS. Vanilla reasoning paradigm. Upper panel: The vanilla reasoning paradigm generates a single, continuous long chain-of-thought in one pass. Lower panel: The InftyThink reasoning paradigm decomposes reasoning into multiple iterative rounds, where consecutive iterations are connected via self-generated global summaries.
• Figure 2: Performance of vanilla reasoning when using InftyThink summaries as context.
• Figure 3: Per-step training time (seconds) over the course of RL training.
• Figure 4: Completion rate and accuracy (%) of vanilla long-context reasoning under different max_new_tokens settings on benchmarks. Dark bars indicate accuracy, while light bars represent the completion rate.
• Figure 5: Computational complexity comparison between vanilla long-context reasoning (blue, left) and InftyThink (pink, right). The sawtooth pattern of InftyThink demonstrates how periodic summarization creates a bounded memory footprint, substantially reducing computational costs (smaller area under curve) while enabling deeper reasoning. We adopt the figure design style from yan2025inftythink.

Theorems & Definitions (6)

• Definition B.1: Optimal Summary
• Proposition B.2: Limitation of Supervised Learning
• proof
• Remark B.3: How RL Addresses These Limitations
• Proposition B.4: Complexity Reduction
• proof