BOSQTGEN: Breaking the Sound Barrier in Test Generation

TL;DR

Problem: API contracts in polyglot ecosystems are brittle, and structured input generation remains a major bottleneck for robust conformance testing. Approach: BosqTGen is a fully black-box test-generation pipeline that decomposes API specifications into primitive components, uses LLMs to partition them into coherent strata, and applies $k$-way covering arrays to sample structured inputs, augmented by a mock-aware data layer. Contributions: (i) contract decomposition with LLM-guided partitions, (ii) lifting combinatorial testing to dynamic/unbounded input domains, (iii) mock-aware test generation, and (iv) comprehensive evaluation showing an average $82\%$ code coverage on five REST benchmarks with up to $20\%$ gains over prior SOTA and near parity with hand-written tests. Significance: enables automatic, API-driven test generation suitable for validation and test-driven development, reducing manual effort and improving fault detection in real-world API-centric development.

Abstract

Modern software is increasingly built by composing APIs, elevating the API contract to a critical role. Inadequate contracts, however, lead to mismatched expectations and failures, creating a pressing need for robust conformance testing. Current test generation techniques are hindered by key challenges: polyglot systems, source code inaccessibility, a cost-reliability trade-off, and, most critically, the difficulty of generating structured inputs. We introduce BOSQTGEN, a novel black-box methodology and tool for API test generation. BOSQTGEN utilizes a novel approach for decomposing API specifications into primitives, using LLMs to suggest coherent strata for them, and employing combinatorial testing to efficiently sample over these values. This approach ensures coverage of critical interactions while avoiding the redundancy of random sampling. The resulting BOSQTGEN system achieves an average of 82% code coverage on RESTful benchmarks, often a 20% or more increase over prior state-of-the-art systems and nearing parity with hand-written test suites. Providing a fully API-driven approach to test generation, enables developers to automatically create high-quality test cases for validation or test-driven development.

Figures (9)

• Figure 1: A sample weather forecast specification in BosqCon.
• Figure 2: A simple combinatorial testing example.
• Figure 3: LLM local context prompt for v.temp.low.value input generation. The {{$\ldots$}} text are where template positions in the prompt have been instantiated with values from the running example. This is concatenated with additional components are appended to produce the final prompt.
• Figure 4: LLM global context prompt for v.temp.low.value input generation. The {{$\ldots$}} text are where template positions in the prompt have been instantiated with values from the running example. This is concatenated with additional components are appended to produce the final prompt.
• Figure 5: LLM mock context prompt for id input generation. This, and the other components from \ref{['sec:llm-partitions']}, are appended to produce the final prompt.