Testora: Using Natural Language Intent to Detect Behavioral Regressions
Michael Pradel
TL;DR
Testora tackles the regression-testing problem by leveraging natural language intent embedded in PRs to distinguish unintended behavioral changes from intended ones. It automates regression detection through a three-part pipeline: targeted LLM-based test generation, differential testing across pre- and post-change builds, and an LLM-based classifier that reasons over PR text to label differences as intended or unintended. Across four large Python projects, Testora identifies 19 real regressions and 11 coincidental fixes, with most regressions later confirmed and addressed by developers, all while maintaining modest costs (~$0.003 per PR, ~12.3 minutes per PR). The approach, implemented in Python with Docker-based isolation, demonstrates practical viability for early regression detection and provides a reproducible test case and natural-language explanations to aid fix validation. Public data and code are released to support further research and adoption.
Abstract
As software is evolving, code changes can introduce regression bugs or affect the behavior in other unintended ways. Traditional regression test generation is impractical for detecting unintended behavioral changes, because it reports all behavioral differences as potential regressions. However, most code changes are intended to change the behavior in some way, e.g., to fix a bug or to add a new feature. This paper presents Testora, the first automated approach that detects regressions by comparing the intentions of a code change against behavioral differences caused by the code change. Given a pull request (PR), Testora queries an LLM to generate tests that exercise the modified code, compares the behavior of the original and modified code, and classifies any behavioral differences as intended or unintended. For the classification, we present an LLM-based technique that leverages the natural language information associated with the PR, such as the title, description, and commit messages -- effectively using the natural language intent to detect behavioral regressions. Applying Testora to PRs of complex and popular Python projects, we find 19 regression bugs and 11 PRs that, despite having another intention, coincidentally fix a bug. Out of 13 regressions reported to the developers, 11 have been confirmed and 9 have already been fixed. The costs of using Testora are acceptable for real-world deployment, with 12.3 minutes to check a PR and LLM costs of only $0.003 per PR. We envision our approach to be used before or shortly after a code change gets merged into a code base, providing a way to early on detect regressions that are not caught by traditional approaches.