Position: Towards a Responsible LLM-empowered Multi-Agent Systems

TL;DR

This paper argues for a principled, responsible LLM-MAS design that explicitly acknowledges and quantifies uncertainty in inter-agent communication and decision-making. It surveys core threats such as knowledge drift, hallucination, collusion, data poisoning, and cyber attacks, and proposes a probabilistic, uncertainty-aware, hierarchical BDI-inspired architecture augmented by a human-centered moderator and runtime provenance. The proposed framework emphasizes formal verification, scalable uncertainty quantification, and dynamic intervention to guarantee system-wide agreement and safety during lifecycle operation. By integrating domain-specific prompting, symbolic rules, and verifiable metrics, the approach aims to deliver reliable, governance-enabled MAS that can safely harness LLM capabilities in complex, real-world tasks.

Abstract

The rise of Agent AI and Large Language Model-powered Multi-Agent Systems (LLM-MAS) has underscored the need for responsible and dependable system operation. Tools like LangChain and Retrieval-Augmented Generation have expanded LLM capabilities, enabling deeper integration into MAS through enhanced knowledge retrieval and reasoning. However, these advancements introduce critical challenges: LLM agents exhibit inherent unpredictability, and uncertainties in their outputs can compound across interactions, threatening system stability. To address these risks, a human-centered design approach with active dynamic moderation is essential. Such an approach enhances traditional passive oversight by facilitating coherent inter-agent communication and effective system governance, allowing MAS to achieve desired outcomes more efficiently.

Figures (7)

• Figure 1: Framework of Reinforcement Learning
• Figure 2: An Illustration for Supervised Fine-tuning
• Figure 3: Framework of Self-improvement
• Figure 4: Cross-Model Agreement Frameworks
• Figure 5: Adversarial Self-Play and Debate Frameworks