Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoning

TL;DR

The paper tackles the inefficiency of long-chain reasoning in large language models by introducing Accordion-Thinking, a framework where models learn to regulate reasoning granularity through periodic, self-generated step summaries and a Fold mode that discards older detailed thoughts. It combines a data synthesis pipeline to train structured stepwise reasoning with reinforcement learning, introducing Unfold, Fold, and Mixed-Mode regimes and a gap-vanishing phenomenon where compressed reasoning eventually matches full-context performance. Empirical results on multiple math benchmarks show Fold-RL and Mix-RL can achieve comparable accuracy to full-context baselines while delivering up to 3x throughput under memory constraints, and the produced step summaries offer readable, faithful insights into the model’s derivations. This approach promises scalable, transparent long-horizon reasoning with practical throughput gains for real-world deployments.

Abstract

Scaling test-time compute via long Chain-ofThought unlocks remarkable gains in reasoning capabilities, yet it faces practical limits due to the linear growth of KV cache and quadratic attention complexity. In this paper, we introduce Accordion-Thinking, an end-to-end framework where LLMs learn to self-regulate the granularity of the reasoning steps through dynamic summarization. This mechanism enables a Fold inference mode, where the model periodically summarizes its thought process and discards former thoughts to reduce dependency on historical tokens. We apply reinforcement learning to incentivize this capability further, uncovering a critical insight: the accuracy gap between the highly efficient Fold mode and the exhaustive Unfold mode progressively narrows and eventually vanishes over the course of training. This phenomenon demonstrates that the model learns to encode essential reasoning information into compact summaries, achieving effective compression of the reasoning context. Our Accordion-Thinker demonstrates that with learned self-compression, LLMs can tackle complex reasoning tasks with minimal dependency token overhead without compromising solution quality, and it achieves a 3x throughput while maintaining accuracy on a 48GB GPU memory configuration, while the structured step summaries provide a human-readable account of the reasoning process.

Figures (5)

• Figure 1: Comparison of Vanilla CoT and our Accordion CoT. As the generation length increases, the computational complexity per token in Vanilla CoT grows quadratically. In contrast, our Accordion CoT folds the context after each step, reducing the computational complexity for the next token generation and improving inference speed. We force the model to follow the Accordion Format, which splits the whole thinking process into several coarse level steps followed by a readable summary. We add 2 special tokens to the model vocabulary. Each generation stops at </step> or the EOS token.
• Figure 2: Ablation study on synthetic Accordion data for Qwen2.5-Math-7B and Qwen3-4B-Base on Fold mode.
• Figure 3: Reward gap between Fold mode and Unfold mode vanishes during Mix-RL training.
• Figure 4: Case study analysis of summary readability. The step summaries, when pieced together, can serve as a substitute for the final solution. Accordion CoT provides users with instant, readable information about the reasoning process.
• Figure 5: Comparison of token efficiency in raw PyTorch.