Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffSpec: Differential Testing with LLMs using Natural Language Specifications and Code Artifacts

Nikitha Rao, Elizabeth Gilbert, Harrison Green, Tahina Ramananandro, Nikhil Swamy, Claire Le Goues, Sarah Fakhoury

TL;DR

This paper tackles differential testing for systems governed by natural language specifications by introducing DiffSpec, a framework that extracts instruction constraints, code snippets, test mappings, and historical bugs from NL docs and code. It leverages prompt chaining with LLMs to generate test descriptions and then executable tests that reveal behavior differences across implementations. Evaluations on eBPF runtimes and Wasm validators show DiffSpec discovers numerous differentiating tests and multiple real bugs, expanding test suites and guiding bug fixes, with generalization demonstrated to Wasm. The results indicate that combining specification and code-context significantly improves the quality and usefulness of differential tests, and point to promising future work in incorporating execution feedback and broadening domain coverage.

Abstract

Differential testing can be an effective way to find bugs in software systems with multiple implementations that conform to the same specification, like compilers, network protocol parsers, or language runtimes. Specifications for such systems are often standardized in natural language documents, like Instruction Set Architecture (ISA) specifications or IETF RFC's. Large Language Models (LLMs) have demonstrated potential in both generating tests and handling large volumes of natural language text, making them well-suited for analyzing artifacts like specification documents, bug reports, and code implementations. In this work, we leverage natural language and code artifacts to guide LLMs to generate targeted tests that highlight meaningful behavioral differences between implementations, including those corresponding to bugs. We introduce DiffSpec, a framework for generating differential tests with LLMs using prompt chaining. We demonstrate DiffSpec's efficacy on two different (extensively tested) systems, eBPF runtimes and Wasm validators. Using DiffSpec, we generated 1901 differentiating tests, uncovering at least four distinct and confirmed bugs in eBPF, including a kernel memory leak, inconsistent behavior in jump instructions, undefined behavior when using the stack pointer, and tests with infinite loops that hang the verifier in ebpf-for-windows. We also found 299 differentiating tests in Wasm validators pointing to two confirmed and fixed bugs.

DiffSpec: Differential Testing with LLMs using Natural Language Specifications and Code Artifacts

TL;DR

This paper tackles differential testing for systems governed by natural language specifications by introducing DiffSpec, a framework that extracts instruction constraints, code snippets, test mappings, and historical bugs from NL docs and code. It leverages prompt chaining with LLMs to generate test descriptions and then executable tests that reveal behavior differences across implementations. Evaluations on eBPF runtimes and Wasm validators show DiffSpec discovers numerous differentiating tests and multiple real bugs, expanding test suites and guiding bug fixes, with generalization demonstrated to Wasm. The results indicate that combining specification and code-context significantly improves the quality and usefulness of differential tests, and point to promising future work in incorporating execution feedback and broadening domain coverage.

Abstract

Differential testing can be an effective way to find bugs in software systems with multiple implementations that conform to the same specification, like compilers, network protocol parsers, or language runtimes. Specifications for such systems are often standardized in natural language documents, like Instruction Set Architecture (ISA) specifications or IETF RFC's. Large Language Models (LLMs) have demonstrated potential in both generating tests and handling large volumes of natural language text, making them well-suited for analyzing artifacts like specification documents, bug reports, and code implementations. In this work, we leverage natural language and code artifacts to guide LLMs to generate targeted tests that highlight meaningful behavioral differences between implementations, including those corresponding to bugs. We introduce DiffSpec, a framework for generating differential tests with LLMs using prompt chaining. We demonstrate DiffSpec's efficacy on two different (extensively tested) systems, eBPF runtimes and Wasm validators. Using DiffSpec, we generated 1901 differentiating tests, uncovering at least four distinct and confirmed bugs in eBPF, including a kernel memory leak, inconsistent behavior in jump instructions, undefined behavior when using the stack pointer, and tests with infinite loops that hang the verifier in ebpf-for-windows. We also found 299 differentiating tests in Wasm validators pointing to two confirmed and fixed bugs.
Paper Structure (20 sections, 3 figures, 5 tables)

This paper contains 20 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Illustrative Example
  3. Approach
  4. Extraction of relevant context from artifacts
  5. Test Generation Framework
  6. Systems Under test
  7. extended Berkeley Packet Filter (eBPF)
  8. WebAssembly (Wasm)
  9. Experimental Design
  10. LLM-based Baselines
  11. Fuzzing Baselines
  12. Evaluation Metrics (RQ1)
  13. Configurations for ablation (RQ2)
  14. Results
  15. RQ1: Overall effectiveness
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Approach overview. DiffSpec extracts relevant context from natural language and code artifacts by prompting an LLM, covering: instructions and constraints, source code and tests, and historical bug information. DiffSpec then follows a two-step process: (1) it generates test descriptions using the extracted context, and then (2) uses the test description along with few-shot examples of human written tests along with a set of human written guidelines to generate actual test code. Generated tests are executed on different implementations of the specification, seeking those that are potentially differentiating.
  • Figure 2: The distribution of generated test complexity (measured by test length) for eBPF. The bug-guided-code-diff (DiffSpec) generates a more complex distribution of tests.
  • Figure 3: Visualization of test status distribution per instruction for eBPF using DiffSpec on Windows.