Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering
Joshua Owotogbe
TL;DR
The paper addresses the robustness of Large Language Model-Based Multi-Agent Systems (LLM-MAS) operating in production-like conditions by proposing a chaos engineering framework to proactively reveal and mitigate vulnerabilities such as communication breakdowns and cascading failures. It adopts a design science methodology to deploy three core contributions: (1) a literature- and repository-driven analysis to identify failure modes (SQ1), (2) a chaos-based testing framework that models and mitigates faults like communication delays and agent failures (SQ2), and (3) an action-research pathway with industry partners to validate robustness and inform certification criteria (SQ3). Initial results include a multivocal literature review and ongoing GitHub repository analysis to map failure modes and guide framework development. The evaluation plan combines quantitative performance metrics with qualitative impact assessments, to be tested in sandboxed and real-world scenarios, and disseminated as open-source tools, aiming for practical impact in industrial LLM-MAS deployments and robustness certification.
Abstract
This study explores the application of chaos engineering to enhance the robustness of Large Language Model-Based Multi-Agent Systems (LLM-MAS) in production-like environments under real-world conditions. LLM-MAS can potentially improve a wide range of tasks, from answering questions and generating content to automating customer support and improving decision-making processes. However, LLM-MAS in production or preproduction environments can be vulnerable to emergent errors or disruptions, such as hallucinations, agent failures, and agent communication failures. This study proposes a chaos engineering framework to proactively identify such vulnerabilities in LLM-MAS, assess and build resilience against them, and ensure reliable performance in critical applications.