Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking Long, but Short: Stable Sequential Test-Time Scaling for Large Reasoning Models

Michael R. Metel, Yufei Cui, Boxing Chen, Prasanna Parthasarathi

TL;DR

This work addresses instability in sequential test-time scaling for large reasoning models by introducing Min-Seek, a training-free method that stabilizes long reasoning chains. It achieves this by retaining only the shortest past reconstruction cycles in a custom KV cache that stores keys without position embeddings, enabling reasoning beyond the model’s maximum context length with linear complexity. Empirically, Min-Seek delivers consistent accuracy gains and robustness across multiple datasets and two distilled DeepSeek-R1 models, outperforming Budget Forcing and offering notable speedups. The approach provides a practical path to unbounded, interpretable long-horizon reasoning in large language models while maintaining efficiency and avoiding the need for context-length fine-tuning.

Abstract

Sequential test-time scaling is a promising training-free method to improve large reasoning model accuracy, but as currently implemented, significant limitations have been observed. Inducing models to think for longer can increase their accuracy, but as the length of reasoning is further extended, it has also been shown to result in accuracy degradation and model instability. This work presents a novel sequential test-time scaling method, Min-Seek, which improves model accuracy significantly over a wide range of induced thoughts, stabilizing the accuracy of sequential scaling, and removing the need for reasoning length fine-tuning. Beyond improving model accuracy over a variety of reasoning tasks, our method is inherently efficient, as only the KV pairs of one additional induced thought are kept in the KV cache during reasoning. With a custom KV cache which stores keys without position embeddings, by dynamically encoding them contiguously before each new generated thought, our method can continue to reason well beyond a model's maximum context length, and under mild conditions has linear computational complexity.

Thinking Long, but Short: Stable Sequential Test-Time Scaling for Large Reasoning Models

TL;DR

This work addresses instability in sequential test-time scaling for large reasoning models by introducing Min-Seek, a training-free method that stabilizes long reasoning chains. It achieves this by retaining only the shortest past reconstruction cycles in a custom KV cache that stores keys without position embeddings, enabling reasoning beyond the model’s maximum context length with linear complexity. Empirically, Min-Seek delivers consistent accuracy gains and robustness across multiple datasets and two distilled DeepSeek-R1 models, outperforming Budget Forcing and offering notable speedups. The approach provides a practical path to unbounded, interpretable long-horizon reasoning in large language models while maintaining efficiency and avoiding the need for context-length fine-tuning.

Abstract

Sequential test-time scaling is a promising training-free method to improve large reasoning model accuracy, but as currently implemented, significant limitations have been observed. Inducing models to think for longer can increase their accuracy, but as the length of reasoning is further extended, it has also been shown to result in accuracy degradation and model instability. This work presents a novel sequential test-time scaling method, Min-Seek, which improves model accuracy significantly over a wide range of induced thoughts, stabilizing the accuracy of sequential scaling, and removing the need for reasoning length fine-tuning. Beyond improving model accuracy over a variety of reasoning tasks, our method is inherently efficient, as only the KV pairs of one additional induced thought are kept in the KV cache during reasoning. With a custom KV cache which stores keys without position embeddings, by dynamically encoding them contiguously before each new generated thought, our method can continue to reason well beyond a model's maximum context length, and under mild conditions has linear computational complexity.
Paper Structure (11 sections, 1 equation, 4 figures)

This paper contains 11 sections, 1 equation, 4 figures.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Test-time Scaling for LLMs
  4. Min-Seek Sequential Test-Time Scaling
  5. KV Cache Management
  6. Selection of Reconstruction Cycles $\overline{RC}_r$
  7. Experiments
  8. Models & Datasets
  9. Implementation Details
  10. Results
  11. Conclusion

Figures (4)

  • Figure 1: Average normalized accuracy over 5 reasoning tasks. An increasing number of thoughts (reconstruction cycles) were induced using Budget Forcing (grey) and our proposed Min-Seek (black) sequential test-time scaling method using DeepSeek-R1-Distill-Qwen-1.5B (above) and DeepSeek-R1-Distill-Qwen-7B (below).
  • Figure 2: An example of Min-Seek while generating a fifth reconstruction cycle (RC 5). The first table shows the position IDs of the prompt and generated tokens, which are grouped together as Thought 1 and RC $i$. The second and third tables show the KV cache without and with our modification before generating RC 2-5, where the boxes, representing the keys and values, are placed according to the keys' position embeddings. The RC $i$ in grey are strictly shorter than all past reconstruction cycles.
  • Figure 3: Average accuracy, from top to bottom, for AIME 2024, AMC, GPQA Diamond, MATH-500, and MMLU-Pro, for an increasing number of induced thoughts (reconstruction cycles) using our proposed method Min-Seek, and Budget Forcing, for DeepSeek-R1-Distill-Qwen-7B (7B) and DeepSeek-R1-Distill-Qwen-1.5B (1.5B).
  • Figure 4: Average normalized computation time over 5 reasoning tasks. A maximum of 0, 10, and 20 thoughts (reconstruction cycles) were induced using our proposed method Min-Seek, and Budget Forcing, for DeepSeek-R1-Distill-Qwen-7B (7B) and DeepSeek-R1-Distill-Qwen-1.5B (1.5B).