Can Past Experience Accelerate LLM Reasoning?

TL;DR

Can past experience accelerate LLM reasoning? The paper formalizes this as a problem of decreasing reasoning cost across sequences of questions with varying similarity and introduces SpeedupLLM, a framework combining adaptive compute budgets and memory. It provides a theoretical analysis proving conditions under which the expected compute budget is non-increasing as exposure grows, and it validates the framework with extensive experiments across memory methods and test-time scaling strategies, reporting up to $56\%$ compute reductions for highly similar questions. The findings show that episodic, in-context memory yields the strongest speedups and that speedup tends to accompany improved accuracy, especially when question similarity is high. The work offers a practical pathway to make LLM reasoning more efficient in real-world, high-throughput settings.

Abstract

Allocating more compute to large language models (LLMs) reasoning has generally been demonstrated to improve their effectiveness, but also results in increased inference time. In contrast, humans can perform tasks faster and better with increased experience and exposure. Hence, this paper aims to investigate the question: Can LLMs also become faster at reasoning through recurrent exposure on relevant tasks, and if so, how can it be achieved? To address these questions, we first formalize the problem setting of LLM reasoning speedup systematically in the dimensions of task relevancy and compute budget calculation. We then propose SpeedupLLM, a theoretically guaranteed framework to implement and benchmark such reasoning speedup behaviour based on adaptive compute allocation and memory mechanisms. We further conduct comprehensive experiments to benchmark such behaviour across different question similarity levels, memory methods, and reasoning methods. Results show that LLMs can generally reason faster with past experience, achieving up to a 56% reduction in compute cost when equipped with appropriate memory and reasoning methods.

Figures (5)

• Figure 1: Changes in compute budget and accuracy relative to the baseline without memory mechanisms, across each memory method, test-time scaling method, and question similarity level. Values represent the relative compute budget and accuracy, averaged over all question backbones and variations.
• Figure 2: Changes in (a) compute budget and (b) accuracy relative to the baseline (no memory), across different memory methods, test-time scaling methods, and question similarity levels. In each subplot, the x-axis denotes the question index within a sequence of questions, while the y-axis shows the percentage change in compute budget or accuracy compared to the baseline. Each curve presents values averaged over multiple question sets at each question index. Gray dashed lines represent baseline (no memory) performance.
• Figure 3: Compute budget changes grouped and averaged by question similarity levels.
• Figure 4: Compute budget changes grouped and averaged by memory methods.
• Figure 5: Compute budget change in each question index, grouped and averaged by memory methods.

Theorems & Definitions (10)

• Theorem 1: Non-Increasing Compute Budget with Non-Decreasing Answer Quality
• proof
• Theorem 2: Non-Decreasing Answer Quality with Accumulating Relevant Memory
• proof
• Corollary 1
• proof
• proof
• proof
• Lemma 1
• proof : Proof of Lemma \ref{['lem:mean2tail']}