SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion

TL;DR

SpecAgent addresses the latency and context-supply gap in repository-aware code completion by shifting context construction to indexing time, enabling rich, speculative cross-file context without adding inference-time latency. The approach combines retrieval and prediction by using indexing-time Retriver, Forecaster, and SpecAgent variants, and introduces an Oracle agent to bound performance; it further mitigates future-context leakage by employing a leakage-free synthetic benchmark. Across two strong models and extensive ablations, SpecAgent yields absolute gains of about 9-11 percentage points (roughly 48-58% relative) over strong baselines while maintaining inference-time latency, demonstrating that offline speculative context can materially improve code-generation quality in large, evolving codebases. The work provides practical significance for IDE-assisted programming, offering a scalable framework for integrating deep repository context into real-time code completion while highlighting the need for realistic benchmarking that avoids leakage.

Abstract

Large Language Models (LLMs) excel at code-related tasks but often struggle in realistic software repositories, where project-specific APIs and cross-file dependencies are crucial. Retrieval-augmented methods mitigate this by injecting repository context at inference time. The low inference-time latency budget affects either retrieval quality or the added latency adversely impacts user experience. We address this limitation with SpecAgent, an agent that improves both latency and code-generation quality by proactively exploring repository files during indexing and constructing speculative context that anticipates future edits in each file. This indexing-time asynchrony allows thorough context computation, masking latency, and the speculative nature of the context improves code-generation quality. Additionally, we identify the problem of future context leakage in existing benchmarks, which can inflate reported performance. To address this, we construct a synthetic, leakage-free benchmark that enables a more realistic evaluation of our agent against baselines. Experiments show that SpecAgent consistently achieves absolute gains of 9-11% (48-58% relative) compared to the best-performing baselines, while significantly reducing inference latency.

Figures (3)

• Figure 1: SpecAgent's workflow at indexing-time (left) involves retrieving relevant contexts and generating target function predictions asynchronously. These are then provided to the code completion model during inference (right).
• Figure 2: Illustration of future context leakage in existing benchmarks. Although the target function save is removed from target_file.py, relevant information remains in the target file and caller_file.py, allowing context retrieval methods to access information that would not exist in a real development scenario.
• Figure 3: The benchmark validation loop.