Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-agent Assisted Automatic Test Generation for Java JSON Libraries

Sinan Wang, Zhiyuan Zhong, Shaojin Wen, Yepang Liu

TL;DR

JsonATG addresses QA gaps for Java JSON libraries by using a multi-agent ATG pipeline tailored to JSON processing. It decomposes test generation into Mutation Rule generation, Test programming, and Test validation, leveraging historical bug-triggering tests and JSON-P/JSON-B documentation. In experiments on fastjson, JsonATG outperformed two baselines in code coverage and bug discovery, reporting 59 bugs with 47 confirmed and 28 fixed. The results show that domain-aware mutation rules, intention summarization, and cross-LLM collaboration yield more diverse, valid tests and uncover both crashing and non-crashing defects, advancing automated QA for JJLs.

Abstract

JSON is a widely used format for data exchange between applications. In the Java ecosystem, JSON libraries serve as fundamental toolkits for processing JSON data, powering real-world applications such as web services, Android apps, or data management systems. However, without effective quality assurance methods such as automatic test generation (ATG), developers risk introducing subtle data inconsistency bugs, compatibility issues, and even security vulnerabilities. These flaws can affect billions of end users and potentially cause severe financial losses. Recently, large language models (LLMs) have shown strong potential in enhancing ATG. However, existing LLM-based methods like TitanFuzz and YanHui lack specialization in the JSON domain. For Java JSON libraries (JJLs), effective test cases should capture the constraints between formatted data and application programs, leaving critical aspects of quality assurance unaddressed. To fill this gap, we propose JsonATG, a multi-agent ATG system that generates diverse bug-triggering tests for JJLs. With historical bug information as seeds, JsonATG introduces a code summarization agent and a test validation agent into the generation pipeline to produce new and valid test cases. It applies agent-generated program mutation rules tailored specifically for the structural and semantic characteristics of JJLs, such as data streaming operations, serialization formats, and data-binding patterns. The generated tests are further refined through post-processing to ensure syntactic and semantic correctness. Our experiments show that JsonATG achieves higher coverage than two state-of-the-art LLM-based test generation methods on the widely used JSON library. With a \$25 budget, we reported 59 bugs (including non-crashing functional bugs) in fastjson, of which 47 were confirmed and 28 have already been fixed.

Multi-agent Assisted Automatic Test Generation for Java JSON Libraries

TL;DR

JsonATG addresses QA gaps for Java JSON libraries by using a multi-agent ATG pipeline tailored to JSON processing. It decomposes test generation into Mutation Rule generation, Test programming, and Test validation, leveraging historical bug-triggering tests and JSON-P/JSON-B documentation. In experiments on fastjson, JsonATG outperformed two baselines in code coverage and bug discovery, reporting 59 bugs with 47 confirmed and 28 fixed. The results show that domain-aware mutation rules, intention summarization, and cross-LLM collaboration yield more diverse, valid tests and uncover both crashing and non-crashing defects, advancing automated QA for JJLs.

Abstract

JSON is a widely used format for data exchange between applications. In the Java ecosystem, JSON libraries serve as fundamental toolkits for processing JSON data, powering real-world applications such as web services, Android apps, or data management systems. However, without effective quality assurance methods such as automatic test generation (ATG), developers risk introducing subtle data inconsistency bugs, compatibility issues, and even security vulnerabilities. These flaws can affect billions of end users and potentially cause severe financial losses. Recently, large language models (LLMs) have shown strong potential in enhancing ATG. However, existing LLM-based methods like TitanFuzz and YanHui lack specialization in the JSON domain. For Java JSON libraries (JJLs), effective test cases should capture the constraints between formatted data and application programs, leaving critical aspects of quality assurance unaddressed. To fill this gap, we propose JsonATG, a multi-agent ATG system that generates diverse bug-triggering tests for JJLs. With historical bug information as seeds, JsonATG introduces a code summarization agent and a test validation agent into the generation pipeline to produce new and valid test cases. It applies agent-generated program mutation rules tailored specifically for the structural and semantic characteristics of JJLs, such as data streaming operations, serialization formats, and data-binding patterns. The generated tests are further refined through post-processing to ensure syntactic and semantic correctness. Our experiments show that JsonATG achieves higher coverage than two state-of-the-art LLM-based test generation methods on the widely used JSON library. With a \$25 budget, we reported 59 bugs (including non-crashing functional bugs) in fastjson, of which 47 were confirmed and 28 have already been fixed.

Paper Structure

This paper contains 37 sections, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Models for Code
  4. Automatic Test Generation
  5. Java JSON Library
  6. JsonATG
  7. The Mutation Rule Generator Agent
  8. The Test Programmer Agent
  9. The Test Validator Agent
  10. Experimental Setup
  11. Subject Library
  12. Bug-triggering Test Collection
  13. Research Questions
  14. Environments
  15. LLM Selection for Agents
  16. ...and 22 more sections

Figures (8)

  • Figure 1: Examples of JSON-P and JSON-B APIs
  • Figure 2: Overview of JsonATG
  • Figure 3: Context of the Mutation Rule Generator agent
  • Figure 4: Context of the Test Programmer agent
  • Figure 5: A bad test modified from the motivating example
  • ...and 3 more figures

Theorems & Definitions (2)

  • Definition 1: Good Test
  • Definition 2: Bad Test