Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpecRover: Code Intent Extraction via LLMs

Haifeng Ruan, Yuntong Zhang, Abhik Roychoudhury

TL;DR

SpecRover tackles the challenge of producing reliable patches from LLMs by introducing iterative specification inference to guide program repair. It extends AutoCodeRover with function-level summaries, reviewer feedback, and evidence to vet and justify patches. The approach yields higher efficacy at modest cost on SWE-bench benchmarks and provides interpretable artifacts to support developer trust. The work underscores the ongoing value of explicit specifications in automated program repair and demonstrates practical deployment potential, including a Linux kernel vulnerability case study.

Abstract

Autonomous program improvement typically involves automatically producing bug fixes and feature additions. Such program improvement can be accomplished by a combination of large language model (LLM) and program analysis capabilities, in the form of an LLM agent. Since program repair or program improvement typically requires a specification of intended behavior - specification inference can be useful for producing high quality program patches. In this work, we examine efficient and low-cost workflows for iterative specification inference within an LLM agent. Given a GitHub issue to be resolved in a software project, our goal is to conduct iterative code search accompanied by specification inference - thereby inferring intent from both the project structure and behavior. The intent thus captured is examined by a reviewer agent with the goal of vetting the patches as well as providing a measure of confidence in the vetted patches. Our approach SpecRover (AutoCodeRover-v2) is built on the open-source LLM agent AutoCodeRover. In an evaluation on the full SWE-Bench consisting of 2294 GitHub issues, it shows more than 50% improvement in efficacy over AutoCodeRover. Compared to the open-source agents available, our work shows modest cost ($0.65 per issue) in resolving an average GitHub issue in SWE-Bench lite. The production of explanation by SpecRover allows for a better "signal" to be given to the developer, on when the suggested patches can be accepted with confidence. SpecRover also seeks to demonstrate the continued importance of specification inference in automated program repair, even as program repair technologies enter the LLM era.

SpecRover: Code Intent Extraction via LLMs

TL;DR

SpecRover tackles the challenge of producing reliable patches from LLMs by introducing iterative specification inference to guide program repair. It extends AutoCodeRover with function-level summaries, reviewer feedback, and evidence to vet and justify patches. The approach yields higher efficacy at modest cost on SWE-bench benchmarks and provides interpretable artifacts to support developer trust. The work underscores the ongoing value of explicit specifications in automated program repair and demonstrates practical deployment potential, including a Linux kernel vulnerability case study.

Abstract

Autonomous program improvement typically involves automatically producing bug fixes and feature additions. Such program improvement can be accomplished by a combination of large language model (LLM) and program analysis capabilities, in the form of an LLM agent. Since program repair or program improvement typically requires a specification of intended behavior - specification inference can be useful for producing high quality program patches. In this work, we examine efficient and low-cost workflows for iterative specification inference within an LLM agent. Given a GitHub issue to be resolved in a software project, our goal is to conduct iterative code search accompanied by specification inference - thereby inferring intent from both the project structure and behavior. The intent thus captured is examined by a reviewer agent with the goal of vetting the patches as well as providing a measure of confidence in the vetted patches. Our approach SpecRover (AutoCodeRover-v2) is built on the open-source LLM agent AutoCodeRover. In an evaluation on the full SWE-Bench consisting of 2294 GitHub issues, it shows more than 50% improvement in efficacy over AutoCodeRover. Compared to the open-source agents available, our work shows modest cost ($0.65 per issue) in resolving an average GitHub issue in SWE-Bench lite. The production of explanation by SpecRover allows for a better "signal" to be given to the developer, on when the suggested patches can be accepted with confidence. SpecRover also seeks to demonstrate the continued importance of specification inference in automated program repair, even as program repair technologies enter the LLM era.
Paper Structure (25 sections, 9 figures, 2 tables)

This paper contains 25 sections, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Methodology
  4. Overview
  5. Function Summary: Specification from Program
  6. Reviewer Feedback: Reconciling Specifications
  7. Patch Selection
  8. Evidence
  9. Experimental Setup
  10. Benchmark
  11. Baselines and Evaluation Metrics
  12. Implementation and Parameters
  13. Randomness of LLMs
  14. Manual Inspection of Results
  15. evaluation
  16. ...and 10 more sections

Figures (9)

  • Figure 1: scikit-learn-15535 description and SpecRover artifacts.
  • Figure 2: Overall Workflow of SpecRover.
  • Figure 3: Context retrieval in SpecRover.
  • Figure 4: Template prompt for the reviewer agent.
  • Figure 5: Number of uniquely resolved issues by the top performing open-source tools on SWE-bench Lite.
  • ...and 4 more figures