VERT: Verified Equivalent Rust Transpilation with Large Language Models as Few-Shot Learners

TL;DR

VERT tackles the challenge of safe, correct transpilation to Rust by integrating a rule-based WebAssembly oracle with an LLM-generated Rust and rigorous equivalence verification. The method iteratively repairs LLM output guided by rustc errors and formal checks (PBT and model-checking) to produce readable, memory-safe Rust code with formal guarantees. Evaluations on 1,394 programs across C++, C, and Go show that Claude-2 in VERT achieves substantial gains in verified transpilations over LLMs alone, and that few-shot repair provides strong benefits. The work also exposes limitations of current LLMs for pointer-heavy Rust and highlights the need for multiple verifiers to scale verification.

Abstract

Rust is a programming language that combines memory safety and low-level control, providing C-like performance while guaranteeing the absence of undefined behaviors by default. Rust's growing popularity has prompted research on safe and correct transpiling of existing code-bases to Rust. Existing work falls into two categories: rule-based and large language model (LLM)-based. While rule-based approaches can theoretically produce correct transpilations that maintain input-output equivalence to the original, they often yield unreadable Rust code that uses unsafe subsets of the Rust language. On the other hand, while LLM-based approaches typically produce more readable, maintainable, and safe code, they do not provide any guarantees about correctness. In this work, we present VERT, a tool that can produce readable Rust transpilations with formal guarantees of correctness. VERT's only requirement is that there is Web Assembly compiler for the source language, which is true for most major languages. VERT first uses the Web Assembly compiler to obtain an oracle Rust program. In parallel, VERT uses an LLM to generate a readable candidate Rust program. This candidate is verified against the oracle, and if verification fails, we regenerate a new candidate transpilation until verification succeeds. We evaluate VERT by transpiling a suite of 1,394 programs taken from competitive programming style benchmarks. Combining Anthropic's Claude-2 and VERT increases Rust transpilations passing property-based testing from 31% to 54% and bounded model-checking from 1% to 42% compared to using Claude alone. In addition, we evaluate VERT's ability to generate non-trivial safe Rust on programs taken from real-world C projects that make significant use of pointers. Our results provide insights into the limitations of LLMs to write safe Rust.

Figures (13)

• Figure 1: Examples of transpilations from a source Go program using compile-then-lift, Anthropic's Claude LLM, and VERT.
• Figure 2: An example bolero harness checking equivalence between 2 functions.
• Figure 3: VERT’s architecture, which takes as input a source program and produces a formally equivalent Rust program
• Figure 4: Syntax error
• Figure 5: Mismatched type error