Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL

TL;DR

RC replaces autoregressive decoding with an iterative, summary-conditioned decoding loop that decouples reasoning horizon from per-turn computation, enabling extrapolation beyond training budgets. It leverages an asymmetry between summarization and generation and employs an off-policy replay mechanism to train models to improve reasoning conditioned on summaries, achieving strong long-horizon performance on math and science benchmarks. Empirically, RC-trained 4B models extrapolate significantly farther than baselines, outperform several larger models on key tasks, and effectively leverage test-time scaffolds to scale inference. The work highlights RC as a general approach to long-horizon reasoning with practical implications for robust, scalable problem solving, while outlining future work on non-myopic rewards and broader domains.

Abstract

Large Language Models (LLMs) that can continually improve beyond their training budgets are able to solve increasingly difficult problems by adapting at test time, a property we refer to as extrapolation. However, standard reinforcement learning (RL) operates over fixed problem distributions and training budgets, which limits extrapolation amidst distribution shift at test time. To address this, we introduce RC, an iterative decoding algorithm that replaces standard autoregressive decoding during both training and inference. RC exploits an asymmetry between the response generation and summarization capabilities of LLMs to construct reasoning chains that consistently improve across iterations. Models trained to use RC can extrapolate and continually improve over reasoning horizons more than an order of magnitude longer than those seen during training. Empirically, training a 4B model with RC using a 16k-token training budget improves performance on HMMT 2025 from 40% to nearly 70% with 0.5m tokens at test time, outperforming both comparably sized models and many larger reasoning LLMs. Finally, we also show that models trained with RC can more effectively leverage existing scaffolds to further scale test-time performance, due to the improved summary-conditioned generation abilities learned through training.

Figures (22)

• Figure 1: Left: Illustration of the RC algorithm.RC decoding replaces standard autoregressive decoding at both train and test time. During RC decoding, the LLM generates a reasoning trace, summarizes it, discards the original trace, and conditions subsequent reasoning on this summary. This design decouples the effective reasoning horizon from the length of any single reasoning trace, thus maintaining tractable rollout lengths for outcome-reward RL while also enabling extrapolation at test time. Right: Performance on HMMT 2025 (November) vs. reasoning token budget. Our RC-trained model RCT-4B (blue, trained from Qwen3-4B-Instruct-2507 at 16k train budget) extrapolates to outperform both the base model with RC decoding (green) and the specialized Qwen3-4B-Thinking-2507 reasoning model (evaluated at 256k test tokens).
• Figure 2: Illustrative example of RC's output. The model generates reasoning given an input problem, before conditioning on the reasoning to generate a summary between 1-2 paragraphs long. The model then conditions on the summary to generate new reasoning. As we show later, RC training improves the ability of the model to perform summary-conditioned reasoning, which enables the model to continually improve over long horizons. See Appendix \ref{['app:add_example']} for a full example of an RC output.
• Figure 3: Left: Accuracy vs. test-time token budget.RC decoding improves performance as token budget $H_\text{test}$ is increased far beyond $H_\text{train} = 16\text{k}$. Middle: Total tokens used vs. test-time token budget. Total reasoning tokens used by RC increases linearly as we increase the reasoning token budget. Shaded regions indicate the 5th-95th percentile; lines indicate the mean. Right: Accuracy vs. token budget for iterative decoding methods.RC is a more effective method for enabling extrapolation than self-verification and self-refinement, highlighting the benefits of exploiting the summarization-generation gap.
• Figure 4: Left: Accuracy at different levels of summary detail. All accuracies measured at $H_\text{test} = 192k$; red dotted line indicates base model performance without RC. Performance degrades when summary detail is either too high or too low. Middle: Relative accuracy improvement (over standard autoregressive decoding). Replacing Qwen3-4B-Instruct-2507 with Qwen3-4B-Thinking-2507 for summarization (Think, Inst) reduces gains. Using Qwen3-4B-Thinking-2507 for both tasks (Think, Think) further reduces gains. Right: Accuracy vs. test-time token budget with various $H_R$. Reducing $H_R$ from 16k to 8k leaves RC performance unchanged, whereas decreasing it further to 4k negatively impacts performance.
• Figure 5: Termination rates of Qwen3-4B-Instruct-2507 with RC as a function of length. Measured on HMMT 2025 and across all $\mathrm{T} = 12$ turns. Virtually all reasoning traces terminate within 16k tokens, suggesting that the model is not trained to reason beyond this. We consider a trace to have terminated after it generates boxed{}.