Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cerberus: Multi-Agent Reasoning and Coverage-Guided Exploration for Static Detection of Runtime Errors

Hridya Dhulipala, Xiaokai Rong, Tien N. Nguyen

TL;DR

Cerberus addresses the challenge of detecting runtime errors in non-executable or incomplete code by introducing an execution-free, coverage-guided testing framework that uses a pair of LLM agents for test-case generation and predictive execution. It employs a two-phase, condition-based prompting strategy to first maximize coverage and then focus on error detection, guided by a predictive executor that estimates coverage and potential exceptions. Across Java and Python, Cerberus demonstrates superior error discovery efficiency and accuracy compared with traditional fuzzers and prior LLM-based methods, including strong performance on incomplete code and real StackOverflow snippets. These results suggest significant potential for pre-execution error detection in code snippets sourced from online forums and educational platforms, with implications for developer tooling and code reliability.

Abstract

In several software development scenarios, it is desirable to detect runtime errors and exceptions in code snippets without actual execution. A typical example is to detect runtime exceptions in online code snippets before integrating them into a codebase. In this paper, we propose Cerberus, a novel predictive, execution-free coverage-guided testing framework. Cerberus uses LLMs to generate the inputs that trigger runtime errors and to perform code coverage prediction and error detection without code execution. With a two-phase feedback loop, Cerberus first aims to both increasing code coverage and detecting runtime errors, then shifts to focus only detecting runtime errors when the coverage reaches 100% or its maximum, enabling it to perform better than prompting the LLMs for both purposes. Our empirical evaluation demonstrates that Cerberus performs better than conventional and learning-based testing frameworks for (in)complete code snippets by generating high-coverage test cases more efficiently, leading to the discovery of more runtime errors.

Cerberus: Multi-Agent Reasoning and Coverage-Guided Exploration for Static Detection of Runtime Errors

TL;DR

Cerberus addresses the challenge of detecting runtime errors in non-executable or incomplete code by introducing an execution-free, coverage-guided testing framework that uses a pair of LLM agents for test-case generation and predictive execution. It employs a two-phase, condition-based prompting strategy to first maximize coverage and then focus on error detection, guided by a predictive executor that estimates coverage and potential exceptions. Across Java and Python, Cerberus demonstrates superior error discovery efficiency and accuracy compared with traditional fuzzers and prior LLM-based methods, including strong performance on incomplete code and real StackOverflow snippets. These results suggest significant potential for pre-execution error detection in code snippets sourced from online forums and educational platforms, with implications for developer tooling and code reliability.

Abstract

In several software development scenarios, it is desirable to detect runtime errors and exceptions in code snippets without actual execution. A typical example is to detect runtime exceptions in online code snippets before integrating them into a codebase. In this paper, we propose Cerberus, a novel predictive, execution-free coverage-guided testing framework. Cerberus uses LLMs to generate the inputs that trigger runtime errors and to perform code coverage prediction and error detection without code execution. With a two-phase feedback loop, Cerberus first aims to both increasing code coverage and detecting runtime errors, then shifts to focus only detecting runtime errors when the coverage reaches 100% or its maximum, enabling it to perform better than prompting the LLMs for both purposes. Our empirical evaluation demonstrates that Cerberus performs better than conventional and learning-based testing frameworks for (in)complete code snippets by generating high-coverage test cases more efficiently, leading to the discovery of more runtime errors.
Paper Structure (48 sections, 9 figures, 8 tables)

This paper contains 48 sections, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Motivation: The Challenges of Runtime Error Detection
  3. Key Ideas and Approach Overview
  4. Cerberus Algorithm
  5. The Multi-Agent Architecture of Cerberus
  6. The Test Case Generation Module
  7. [Phase 1] Prompting for Test Case Generation to Increase Coverage and Detect Runtime Errors
  8. [Phase 2] Prompting for Test Case Generation towards Runtime Error Detection
  9. LLM-based Code Coverage Prediction
  10. Empirical Evaluation
  11. Runtime Error Detection on Java Code Snippets (RQ1)
  12. Baselines, Dataset, Procedure, and Metrics
  13. Baselines
  14. Dataset
  15. Procedure
  16. ...and 33 more sections

Figures (9)

  • Figure 1: Predictive Coverage-Guided Testing with Feedback Loop and Condition-based Dual Prompting
  • Figure 2: An Example of a Target Java Code
  • Figure 3: Cerberus Algorithm
  • Figure 4: Portion of prompt to generate test cases to 'increase coverage'
  • Figure 5: Portion of prompt to generate test cases intended to 'catch runtime errors'
  • ...and 4 more figures