Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fixing Rust Compilation Errors using LLMs

Pantazis Deligiannis, Akash Lal, Nikita Mehrotra, Aseem Rastogi

TL;DR

RustAssistant demonstrates that Large Language Models can be effectively used to automatically fix real-world Rust compilation errors by iteratively prompting the model and validating changes against the compiler. The approach combines error localization, a specialized changelog-based patch format, and a compiler-driven feedback loop, with GPT-4 delivering substantially higher fix rates than GPT-3.5 across micro-benchmarks, Stack Overflow, and top crates, while also generalizing to Clippy lint errors. The work provides a comprehensive Rust error dataset and shows strong practical potential for LLM-assisted software maintenance, backed by thorough ablation studies and threat analyses. Open-sourcing the dataset and toolchain will enable broader validation and future enhancements in LLM-driven automated program repair for Rust.

Abstract

The Rust programming language, with its safety guarantees, has established itself as a viable choice for low-level systems programming language over the traditional, unsafe alternatives like C/C++. These guarantees come from a strong ownership-based type system, as well as primitive support for features like closures, pattern matching, etc., that make the code more concise and amenable to reasoning. These unique Rust features also pose a steep learning curve for programmers. This paper presents a tool called RustAssistant that leverages the emergent capabilities of Large Language Models (LLMs) to automatically suggest fixes for Rust compilation errors. RustAssistant uses a careful combination of prompting techniques as well as iteration with an LLM to deliver high accuracy of fixes. RustAssistant is able to achieve an impressive peak accuracy of roughly 74% on real-world compilation errors in popular open-source Rust repositories. We plan to release our dataset of Rust compilation errors to enable further research.

Fixing Rust Compilation Errors using LLMs

TL;DR

RustAssistant demonstrates that Large Language Models can be effectively used to automatically fix real-world Rust compilation errors by iteratively prompting the model and validating changes against the compiler. The approach combines error localization, a specialized changelog-based patch format, and a compiler-driven feedback loop, with GPT-4 delivering substantially higher fix rates than GPT-3.5 across micro-benchmarks, Stack Overflow, and top crates, while also generalizing to Clippy lint errors. The work provides a comprehensive Rust error dataset and shows strong practical potential for LLM-assisted software maintenance, backed by thorough ablation studies and threat analyses. Open-sourcing the dataset and toolchain will enable broader validation and future enhancements in LLM-driven automated program repair for Rust.

Abstract

The Rust programming language, with its safety guarantees, has established itself as a viable choice for low-level systems programming language over the traditional, unsafe alternatives like C/C++. These guarantees come from a strong ownership-based type system, as well as primitive support for features like closures, pattern matching, etc., that make the code more concise and amenable to reasoning. These unique Rust features also pose a steep learning curve for programmers. This paper presents a tool called RustAssistant that leverages the emergent capabilities of Large Language Models (LLMs) to automatically suggest fixes for Rust compilation errors. RustAssistant uses a careful combination of prompting techniques as well as iteration with an LLM to deliver high accuracy of fixes. RustAssistant is able to achieve an impressive peak accuracy of roughly 74% on real-world compilation errors in popular open-source Rust repositories. We plan to release our dataset of Rust compilation errors to enable further research.
Paper Structure (18 sections, 7 figures, 7 tables, 1 algorithm)

This paper contains 18 sections, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Overview
  4. Scope
  5. Overview of the RustAssistant pipeline
  6. RustAssistant Implementation
  7. Inner loop for fixing an input error
  8. Prompt construction (instantiate_prompt)
  9. LLM invocation (invoke_llm)
  10. Rust Error Dataset
  11. Evaluation
  12. LLM Configuration
  13. Summary.
  14. Threats to Validity
  15. Internal Threats
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Snippet from a Stack Overflow question
  • Figure 2: Output of RustAssistant.
  • Figure 3: Error after the first fix suggested by RustAssistant
  • Figure 4: The RustAssistant prompt template.
  • Figure 5: Comparing the fix rate % of GPT-3.5 and GPT-4 (both with $N=5$) across error code categories.
  • ...and 2 more figures