On The Importance of Reasoning for Context Retrieval in Repository-Level Code Editing

TL;DR

This paper probes context retrieval for repository-level code editing by decoupling retrieval from downstream editing and systematically evaluating retrieval components. It compares BM25 baselines with ReAct-style, tool-enhanced agents across two datasets using localization metrics to measure how well retrieved context aligns with changes. The results indicate that reasoning boosts precision in gathered context, though recall is mainly governed by the amount of context available, and that specialized tools substantially improve retrieval performance. The findings inform the design of retrieval-focused modules and agent interfaces, highlighting directions for future cross-LLM studies and broader method evaluations.

Abstract

Recent advancements in code-fluent Large Language Models (LLMs) enabled the research on repository-level code editing. In such tasks, the model navigates and modifies the entire codebase of a project according to request. Hence, such tasks require efficient context retrieval, i.e., navigating vast codebases to gather relevant context. Despite the recognized importance of context retrieval, existing studies tend to approach repository-level coding tasks in an end-to-end manner, rendering the impact of individual components within these complicated systems unclear. In this work, we decouple the task of context retrieval from the other components of the repository-level code editing pipelines. We lay the groundwork to define the strengths and weaknesses of this component and the role that reasoning plays in it by conducting experiments that focus solely on context retrieval. We conclude that while the reasoning helps to improve the precision of the gathered context, it still lacks the ability to identify its sufficiency. We also outline the ultimate role of the specialized tools in the process of context gathering. The code supplementing this paper is available at https://github.com/JetBrains-Research/ai-agents-code-editing.