CoverUp: Effective High Coverage Test Generation for Python

TL;DR

CoverUp addresses the challenge of generating high-coverage Python regression tests by combining coverage analysis, code context, and iterative prompts to an LLM. Its pipeline segments code around coverage gaps, prompts the LLM with targeted context, executes generated tests, and refines prompts through feedback, including a get_info tool for additional code context. Empirical results show CoverUp outperforms state-of-the-art baselines (CodaMosa and MuTAP) on multiple benchmarks, with significant gains in line and branch coverage and a meaningful contribution from continued dialogues. Ablation studies indicate that coverage-based prompting, code context, and error-aware feedback are all important to achieving high coverage, suggesting practical impact for scalable regression testing in Python projects.

Abstract

Testing is an essential part of software development. Test generation tools attempt to automate the otherwise labor-intensive task of test creation, but generating high-coverage tests remains challenging. This paper proposes CoverUp, a novel approach to driving the generation of high-coverage Python regression tests. CoverUp combines coverage analysis, code context, and feedback in prompts that iteratively guide the LLM to generate tests that improve line and branch coverage. We evaluate our prototype CoverUp implementation across a benchmark of challenging code derived from open-source Python projects and show that CoverUp substantially improves on the state of the art. Compared to CodaMosa, a hybrid search/LLM-based test generator, CoverUp achieves a per-module median line+branch coverage of 80% (vs. 47%). Compared to MuTAP, a mutation- and LLM-based test generator, CoverUp achieves an overall line+branch coverage of 89% (vs. 77%). We also demonstrate that CoverUp's performance stems not only from the LLM used but from the combined effectiveness of its components.

Figures (18)

• Figure 1: Graphical Overview of the CoverUp Algorithm.After measuring the test suite's coverage, CoverUp prompts the LLM, focusing it on code segments that lack coverage. It checks each new test, either accepting it if it increases coverage or continuing the dialogue if it needs improvements. CoverUp provides a tool function with which the LLM can request additional context. A final check helps ensure that the suite works as a whole. Given its incremental nature, once done, CoverUp can be rerun to refocus the LLM on any segments missed in previous passes.
• Figure 2: CoverUp summarizes method excerpts:CoverUp generates compact code excerpts. In this case, 200 lines of source code were originally present between listing lines 1 and 2. The original code is from the flutils package.
• Figure 3: Example of an initial prompt:The initial prompt selects a persona (1), identifies where the code comes from and states how it lacks coverage when tested (2), asks for tests (3-5), and shows the code segment, prefixed by import statements and with the line lacking coverage identified by its number. The code shown is from the thonny package.
• Figure 4: Sending a prompt: to prompt and LLM using OpenAI's chat API, CoverUp puts together a JSON-formatted request that includes the prompt in "messages". [...] indicates a portion omitted for brevity.
• Figure 5: Continuing the chat: to continue a chat, CoverUp sends out a new request, including the previous messages (here, the initial prompt), the LLM's response (indicated with the "assistant" role), and the new prompt. [...] indicates a portion omitted for brevity.