Multi-Level Testing of Conversational AI Systems
Elena Masserini
TL;DR
The paper addresses quality assurance for conversational AI, where non-determinism, natural language variability, and multi-component interactions hinder traditional testing. It proposes a three-level testing framework—service-interaction, agent, and multi-agent system testing—coupled with search-based test generation, feedback-directed gray-box strategies, and LLM-assisted exploration, augmented by metamorphic testing and planning/orchestration for multi-agent workflows. The anticipated contributions include a curated dataset of conversational systems, novel testing methodologies across granularity levels, testing/mocking agents, and empirical evaluation demonstrating improved fault detection and reliability. This framework aims to enable more trustworthy, scalable QA for single and multi-agent conversational AI deployments across domains.
Abstract
Conversational AI systems combine AI-based solutions with the flexibility of conversational interfaces. However, most existing testing solutions do not straightforwardly adapt to the characteristics of conversational interaction or to the behavior of AI components. To address this limitation, this Ph.D. thesis investigates a new family of testing approaches for conversational AI systems, focusing on the validation of their constituent elements at different levels of granularity, from the integration between the language and the AI components, to individual conversational agents, up to multi-agent implementations of conversational AI systems