Interleaved Reasoning for Large Language Models via Reinforcement Learning

TL;DR

This work tackles the inefficiency of long chain-of-thought reasoning in large language models by introducing interleaved reasoning, where thinking and answering alternate during multi-hop tasks. The authors implement a reinforcement learning framework with a simple, rule-based reward design that includes a format reward, a final accuracy reward, and conditional intermediate rewards, guided by an interleaved <think>/<answer> template. Across five datasets and three RL algorithms, the approach achieves a 12.5% relative improvement in Pass@1 and reduces TTFT by over 80%, while also shortening reasoning length by up to 37%, demonstrating strong generalization to unseen tasks like MATH, GPQA, and MMLU. The findings offer practical, tool-free improvements in reasoning efficiency and credit assignment for LLMs, with broader implications for interactive AI systems and deployment in real-time scenarios.

Abstract

Long chain-of-thought (CoT) significantly enhances the reasoning capabilities of large language models (LLMs). However, extensive reasoning traces lead to inefficiencies and increased time-to-first-token (TTFT). We propose a training paradigm that uses only reinforcement learning (RL) to guide reasoning LLMs to interleave thinking and answering for multi-hop questions. We observe that models inherently possess the ability to perform interleaved reasoning, which can be further enhanced through RL. We introduce a simple yet effective reward scheme to incentivize correct intermediate steps, guiding the policy model toward correct reasoning paths by leveraging intermediate signals generated during interleaved reasoning. Extensive experiments across five diverse datasets and three RL algorithms (PPO, GRPO, and REINFORCE++) demonstrate consistent improvements over traditional think-answer reasoning, without requiring external tools. Our method improves final task accuracy and overall efficiency by enabling more effective credit assignment during RL. Specifically, our approach achieves a 12.5% improvement in Pass@1 accuracy, while reducing overall reasoning length by 37% and TTFT by over 80% on average. Furthermore, our method, trained solely on question answering and logical reasoning datasets, exhibits strong generalization to complex reasoning datasets such as MATH, GPQA, and MMLU. Additionally, we conduct in-depth analysis to reveal several valuable insights into conditional reward modeling.

Figures (4)

• Figure 1: Standard think-answer reasoning (left) completes the full chain-of-thought before generating an answer, resulting in high TTFT and making credit assignment difficult during training when intermediate steps contain errors (highlighted in yellow). Interleaved reasoning (right) alternates between thinking and answering, enabling structured, easy-to-verify reward signals for better credit assignment and significantly reducing TTFT.
• Figure 2: The format reward plateaus rapidly during training, significantly faster than the accuracy reward, suggesting that LLMs inherently possess the ability to reason in structured, interleaved patterns.
• Figure 3: Analysis of interleaved reasoning: (a) Pass@1 accuracy vs. problem difficulty: performance gap widens as K&K problem difficulty increases; (b) Training dynamics: convergence patterns across different RL algorithms; (c) Response length vs. correctness: correct answers are typically shorter across all methods; (d) Intermediate reward impact: percentage of correct intermediate answers increases with conditional intermediate rewards.
• Figure 4: Visualization of intermediate reward application rate during training. The rate decreases as training progresses due to increasing batch accuracy thresholds.