Structured Reasoning for Large Language Models

TL;DR

The paper addresses inefficiencies in unstructured long-chain reasoning in LLMs by introducing Structured Reasoning (SCR), which decouples generation, verification, and revision into a verifiable trajectory. It implements Generate–Verify–Revise with Dynamic Termination Supervision and a two-stage reinforcement learning scheme (Stage I for generation and self-verification, Stage II for revision) to train each ability independently, using a targeted data-synthesis and masking strategy. Across three backbone models, SCR achieves higher reasoning performance, stronger self-verification, and up to a 50% reduction in output tokens, demonstrating better efficiency and robustness on math and general benchmarks. The approach broadens practical impact by improving reasoning reliability and generalization beyond math domains, suggesting a viable path for more structured and controllable LLM reasoning.

Abstract

Large language models (LLMs) achieve strong performance by generating long chains of thought, but longer traces always introduce redundant or ineffective reasoning steps. One typical behavior is that they often perform unnecessary verification and revisions even if they have reached the correct answers. This limitation stems from the unstructured nature of reasoning trajectories and the lack of targeted supervision for critical reasoning abilities. To address this, we propose Structured Reasoning (SCR), a framework that decouples reasoning trajectories into explicit, evaluable, and trainable components. We mainly implement SCR using a Generate-Verify-Revise paradigm. Specifically, we construct structured training data and apply Dynamic Termination Supervision to guide the model in deciding when to terminate reasoning. To avoid interference between learning signals for different reasoning abilities, we adopt a progressive two-stage reinforcement learning strategy: the first stage targets initial generation and self-verification, and the second stage focuses on revision. Extensive experiments on three backbone models show that SCR substantially improves reasoning efficiency and self-verification. Besides, compared with existing reasoning paradigms, it reduces output token length by up to 50%.

Figures (7)

• Figure 1: Comparison of reasoning trajectories.Long CoT: A single reasoning process in which all thinking operators are blended together, resulting in redundant self-verification loops even after a correct solution is reached. Ours: A structured reasoning trajectory with explicitly separated components, where each stage is verifiable and trainable. Generation stops once correctness is confirmed.
• Figure 2: Behavioral analysis of long-chain reasoning in large reasoning models.Left: Frequency distribution of different thinking operators within long CoT trajectories. Right: Effectiveness of revision behaviors in long CoT, quantified by the transition in answer correctness pre- and post-revision.
• Figure 3: Overview of the proposed training framework.Left: SFT with Dynamic Termination Supervision and Selective Loss Masking. Right: Staged reinforcement learning with stage-specific reward designs for learning initial generation, self-verification, and revision.
• Figure 4: Ablation of DTS and SLM during the SFT phase. Removing DTS causes a sharp performance drop on hard benchmarks, while removing SLM leads to consistent degradation across tasks.
• Figure 5: Performance comparison between staged RL training and unified mixed optimization. Progressive staged RL training leads to consistently better performance than unified mixed optimization, and the second stage further strengthens model performance.