Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons

TL;DR

Breadcrumbs Reasoning introduces a memory-efficient mechanism for long-context reasoning by periodically compressing KV-cache entries into beacon tokens, enabling substantial test-time scaling under tight memory budgets. The method employs a joint RL-distillation framework in which a non-compressing RL teacher guides the training of a compression-enabled BR policy, achieving a favorable memory-accuracy Pareto frontier. Empirical results on Qwen and Phi models across Countdown, LinSys, and StarGraph show BR can maintain most of uncompressed performance while reducing memory usage by factors up to $32\times$, with multi-ratio training offering flexible deployment. The work highlights the necessity of learned compression for coherent long-horizon reasoning and outlines future directions in dynamic ratio selection and integration with existing CoT strategies to further enhance efficiency and generalization.

Abstract

The scalability of large language models for long-context reasoning is severely constrained by the linear growth of their Transformer key-value cache, which incurs significant memory and computational costs. We posit that as a model generates reasoning tokens, the informational value of past generated tokens diminishes, creating an opportunity for compression. In this work, we propose to periodically compress the generation KV cache with a learned, special-purpose token and evict compressed entries. We train the model to perform this compression via a modified joint distillation and reinforcement learning (RL) framework. Our training method minimizes overhead over the conventional RL process, as it leverages RL outputs for distillation. Empirically, our method achieves a superior memory-accuracy Pareto frontier compared to both the model without cache compression and training-free compression techniques.

Figures (7)

• Figure 1: Breadcrumbs Reasoning, with a compression ratio $c=16$. To save memory during inference, a window of $c$ tokens is periodically compressed into a single beacon token. The original KV cache entries for the window are then evicted, leaving only a compact 'breadcrumb' that summarizes the preceding reasoning steps.
• Figure 2: Attention mask used to enforce compression during training. Each token after the question can attend to the initial question tokens, all previous beacons, and earlier generation tokens within the same window, i.e., since the most recent preceding beacon. This encourages the model to compress relevant past context into the beacons to support future generations.
• Figure 3: Accuracy vs. KV Cache Size Multi Ratio Breadcrumbs Reasoning retains most of the teacher's performance while using significantly fewer KV cache entries, even outperforming the teacher when setting a fixed maximum KV cache size in Countdown and StarGraph.
• Figure 4: Failure mode analysis on LinSys. We classify error types of Qwen with SR BR across increasing compression ratios. The results indicate that the performance drop in LinSys is primarily driven by arithmetic errors, which increase significantly with compression, rather than a failure to memorize or retrieve information.
• Figure 5: Joint RL-distillation vs. Two-step Training on Qwen. We compare our joint approach to the standard two-step process, where compression policies are trained on trajectories from a final $\pi_{\rm RL}$ checkpoint. Each point shows the accuracy difference (Joint $-$ Two-step) at a given compression ratio. The top row fixes cache size; the bottom fixes response length. Positive values favor joint training, which outperforms or matches in 26/30 settings, showing BR can learn compression online during RL without separate distillation data.