Interpreting Emergent Extreme Events in Multi-Agent Systems

TL;DR

The paper tackles the interpretability of emergent extreme events in multi-agent systems by attributing final risk to individual agent actions via a faithful Shapley-value decomposition across time, agents, and behaviors. It formulates precise definitions of when events originate, who drives them, and which behaviors contribute, and introduces time-aware and multi-dimensional metrics to characterize event features. Empirical results across economic, financial, and social simulations show the method accurately attributes risk, identifies small cohorts of high-impact agents and behaviors, and outperforms baselines in faithfulness. The approach offers actionable insight for safety and risk management in complex MAS and demonstrates how counterfactual reasoning can translate into robust attribution under fat-tail dynamics.

Abstract

Large language model-powered multi-agent systems have emerged as powerful tools for simulating complex human-like systems. The interactions within these systems often lead to extreme events whose origins remain obscured by the black box of emergence. Interpreting these events is critical for system safety. This paper proposes the first framework for explaining emergent extreme events in multi-agent systems, aiming to answer three fundamental questions: When does the event originate? Who drives it? And what behaviors contribute to it? Specifically, we adapt the Shapley value to faithfully attribute the occurrence of extreme events to each action taken by agents at different time steps, i.e., assigning an attribution score to the action to measure its influence on the event. We then aggregate the attribution scores along the dimensions of time, agent, and behavior to quantify the risk contribution of each dimension. Finally, we design a set of metrics based on these contribution scores to characterize the features of extreme events. Experiments across diverse multi-agent system scenarios (economic, financial, and social) demonstrate the effectiveness of our framework and provide general insights into the emergence of extreme phenomena.

Figures (5)

• Figure 1: Overview of the proposed framework for explaining Black Swans (emergent extreme events) in MAS. (a) Black Swans are defined as statistical outliers in the fat tails of risk distributions, carrying extreme systemic impact and being explainable only after the occurrence. (b) We explain extreme events by answering the questions from the perspective of when, who, and what. Specifically, we use attribution methods to quantify the risk contributions across these three dimensions. Based on these contributions, we design a set of metrics to quantitatively analyze the features of extreme events.
• Figure 2: Risk attribution process and quantitative metric derivation. Step 1: action-level attribution. We first attribute the extreme event to all individual actions $a_{i,t}$ in the trajectory leading to the event, assigning each an attribution score $\phi(a_{i,t})$. Red indicates actions that increase the risk of the extreme event, while blue represents actions that decrease it. Step 2-4: dimensional aggregation. These attribution scores are subsequently aggregated along three dimensions of time $\phi^{\text{tm}}_{t}$, agent $\phi^{\text{ag}}_{i}$, and behavior $\phi^{\text{be}}_{k}$ to quantify their respective contributions. Metrics. Based on these aggregated contributions, we derive five quantitative metrics to characterize the extreme event: relative risk latency $L_{\text{tm}}$, agent risk concentration $G_{\text{ag}}$, risk-instability correlation $C_{\text{ag}}$, agent risk synchronization $Z_{\text{ag}}$, and behavioral risk concentration $G_{\text{be}}$.
• Figure 3: Visualization of risk latency. The bar chart represents the risk contribution $\phi^{\text{tm}}_{t}$ of all actions taken at time $t$. The red line denotes the cumulative risk $\sum_{t'=1}^{t}\phi^{\text{tm}}_{t'}$, while the blue line represents the observed risk $R_{t}(\tau)$ in the real world at time $t$.
• Figure 4: Visualization of risk from each agent. The agents are sorted by the value of $\phi_i^{\text{ag}}$. (a) Correlation between agent instability and absolute risk contribution ($C_{\text{ag}} = 0.61$). (b) Distribution of risk contributions $\phi^{\text{ag}}_i$ across agents, showing significant risk concentration ($G_{\text{ag}} = 0.54$). (c) Configuration prompts for high-risk (agent 1, short-term) and low-risk (agent 20, long-term) investment strategies.
• Figure 5: Visualization of the risk contribution $\phi^{\text{be}}_{k}$ of different behaviors $\mathcal{A}_k$. The contributions are normalized such that the total equals 100%. We also highlight the high-risk behaviors in each system. GPT, Llama, Claude, Qwen, and DS denote the results of model GPT-4o mini, Llama-3.1-8B-Instruct, Claude-3-Haiku, Qwen-Plus, and DeepSeek-V3.2, respectively.

Theorems & Definitions (10)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• proof
• proof
• proof
• proof
• proof