ResMAS: Resilience Optimization in LLM-based Multi-agent Systems
Zhilun Zhou, Zihan Liu, Jiahe Liu, Qingyu Shao, Yihan Wang, Kun Shao, Depeng Jin, Fengli Xu
TL;DR
ResMAS addresses resilience in LLM-based MAS under random perturbations by jointly optimizing topology and prompts. It introduces a two-stage framework where a topology generator (fine-tuned LLM) is guided by a graph neural network reward model and GRPO to produce resilient architectures, followed by topology-aware prompt optimization that tailors each agent's instructions to its neighborhood. The resilience measure $R(\mathcal{G})=\frac{1}{F(0)}\int_{0}^{1}F(p)\,dp$ (approximated discretely as $R(\mathcal{G})=\frac{1}{10F(0)}(F(0)+2F(0.2)+2F(0.4)+2F(0.6)+2F(0.8)+F(1))$) guides optimization; experiments on MATH, MMLU-Pro, Chess, and HumanEval demonstrate improved resilience and strong generalization across tasks and backbones. The approach yields superior resilience across datasets, reveals a clear benefit from decentralized, evenly connected topologies, and shows that topology-aware prompting significantly enhances robustness. By enabling joint optimization of structure and communication-aware prompts, ResMAS offers practical means to deploy robust, scalable MAS in distributed and potentially heterogeneous environments.
Abstract
Large Language Model-based Multi-Agent Systems (LLM-based MAS), where multiple LLM agents collaborate to solve complex tasks, have shown impressive performance in many areas. However, MAS are typically distributed across different devices or environments, making them vulnerable to perturbations such as agent failures. While existing works have studied the adversarial attacks and corresponding defense strategies, they mainly focus on reactively detecting and mitigating attacks after they occur rather than proactively designing inherently resilient systems. In this work, we study the resilience of LLM-based MAS under perturbations and find that both the communication topology and prompt design significantly influence system resilience. Motivated by these findings, we propose ResMAS: a two-stage framework for enhancing MAS resilience. First, we train a reward model to predict the MAS's resilience, based on which we train a topology generator to automatically design resilient topology for specific tasks through reinforcement learning. Second, we introduce a topology-aware prompt optimization method that refines each agent's prompt based on its connections and interactions with other agents. Extensive experiments across a range of tasks show that our approach substantially improves MAS resilience under various constraints. Moreover, our framework demonstrates strong generalization ability to new tasks and models, highlighting its potential for building resilient MASs.