Automated structural testing of LLM-based agents: methods, framework, and case studies

TL;DR

This paper tackles the challenge of testing LLM-based agents, which are prone to non-determinism and internal failures, by introducing structural testing techniques that reveal internal behavior. It proposes three technical pillars—OpenTelemetry traces, LLM mocking, and automated assertions—to enable glass-box testing of agent components and their interactions. The authors adapt software engineering practices such as the test automation pyramid and test-driven development to agents and validate the approach with case studies (driver assistance and cloud RCA), including automated test execution and faster root-cause analysis. An open-source reference implementation is released to encourage adoption and cross-platform migration.

Abstract

LLM-based agents are rapidly being adopted across diverse domains. Since they interact with users without supervision, they must be tested extensively. Current testing approaches focus on acceptance-level evaluation from the user's perspective. While intuitive, these tests require manual evaluation, are difficult to automate, do not facilitate root cause analysis, and incur expensive test environments. In this paper, we present methods to enable structural testing of LLM-based agents. Our approach utilizes traces (based on OpenTelemetry) to capture agent trajectories, employs mocking to enforce reproducible LLM behavior, and adds assertions to automate test verification. This enables testing agent components and interactions at a deeper technical level within automated workflows. We demonstrate how structural testing enables the adaptation of software engineering best practices to agents, including the test automation pyramid, regression testing, test-driven development, and multi-language testing. In representative case studies, we demonstrate automated execution and faster root-cause analysis. Collectively, these methods reduce testing costs and improve agent quality through higher coverage, reusability, and earlier defect detection. We provide an open source reference implementation on GitHub.

Figures (5)

• Figure 1: Structure of an LLM-based agent. Source: based on xi2023risepotentiallargelanguage
• Figure 2: Test framework for LLM-based agents.
• Figure 3: Simplified architectural overview for driver assistance agent.
• Figure 4: screenshot of driver assistance agent's test coverage
• Figure 5: Architectural overview for root cause analysis agent. Source: RCA-agent