PyResBugs: A Dataset of Residual Python Bugs for Natural Language-Driven Fault Injection

TL;DR

PyResBugs tackles the challenge of representing residual faults in Python software by curating $5{,}007$ fault pairs from real-world projects, each linked to a fixed version and enriched with three levels of natural-language fault descriptions. By combining method-level fault extraction, CVE-linked data, and a comprehensive metadata schema, the dataset enables NL-driven fault injection and AI-assisted testing that better reflects production faults. The authors extend the Orthogonal Defect Classification to Python-specific residual faults and demonstrate a broad fault diversity, with detailed NL descriptions to support diverse prompting strategies. Public availability on GitHub and the intention to drive an SFI tool illustrate PyResBugs' potential to advance realistic, accessible, and automated fault injection in Python ecosystems.

Abstract

This paper presents PyResBugs, a curated dataset of residual bugs, i.e., defects that persist undetected during traditional testing but later surface in production, collected from major Python frameworks. Each bug in the dataset is paired with its corresponding fault-free (fixed) version and annotated with multi-level natural language (NL) descriptions. These NL descriptions enable natural language-driven fault injection, offering a novel approach to simulating real-world faults in software systems. By bridging the gap between software fault injection techniques and real-world representativeness, PyResBugs provides researchers with a high-quality resource for advancing AI-driven automated testing in Python systems.

Figures (1)

• Figure 1: Example of a deadlock fault in CPython: faulty and fault-free code versions, with three NL descriptions for bug injection in the put method