Causal Explanations for Sequential Decision-Making in Multi-Agent Systems

TL;DR

CEMA presents a general framework for generating causal natural language explanations for ego-agent decisions in dynamic multi-agent systems by exploiting probabilistic forward models to simulate counterfactual worlds. It operationalizes causal selection via the Counterfactual Effect Size Model and distinguishes teleological and mechanistic explanations without relying on fixed structural causal graphs. The approach is demonstrated in autonomous driving motion planning, including implementation details with a defined feature set, forward simulations, and deterministic natural language realization, plus a large user study and the HEADD dataset. The work shows that CEMA can robustly identify salient causes across many agents and improves perceived trust in autonomous vehicles, offering a practical path toward trustworthy social XAI in sequential multi-agent settings.

Abstract

We present CEMA: Causal Explanations in Multi-Agent systems; a framework for creating causal natural language explanations of an agent's decisions in dynamic sequential multi-agent systems to build more trustworthy autonomous agents. Unlike prior work that assumes a fixed causal structure, CEMA only requires a probabilistic model for forward-simulating the state of the system. Using such a model, CEMA simulates counterfactual worlds that identify the salient causes behind the agent's decisions. We evaluate CEMA on the task of motion planning for autonomous driving and test it in diverse simulated scenarios. We show that CEMA correctly and robustly identifies the causes behind the agent's decisions, even when a large number of other agents is present, and show via a user study that CEMA's explanations have a positive effect on participants' trust in autonomous vehicles and are rated as high as high-quality baseline explanations elicited from other participants. We release the collected explanations with annotations as the HEADD dataset.

Figures (24)

• Figure 1: The autonomous vehicle ($\varepsilon$) is heading to the blue goal. It decided to change lanes after the other vehicle (1) cut in front of it and began to slow down. A passenger asks: Why did you change lanes? "To decrease the time to reach the goal." [teleological] Why was changing lanes faster? "Because the other vehicle is slower than us and is decelerating." [mechanistic] -- Actual explanations by CEMA with explanation types in brackets. Blue/orange lines illustrate forward simulations using the probabilistic forward model.
• Figure 2: First, irrelevant observations are filtered out based on the query. Second, CEMA rolls back the filtered observations to a previous timestep so that the queried action is erased. From then, CEMA simulates counterfactual worlds to calculate the counterfactual causal effect size for the queried actions, which are used to rank the features of the system.
• Figure 3: The four scenarios used for evaluation based on albrechtInterpretableGoalbasedPrediction2021. Colored circles are goals. Solid lines are predicted trajectories of non-egos with thickness corresponding to predicted probability. Black dotted lines are observations.
• Figure 4: [Top] Signed differences between expected reward components correctly identify time-to-goal as the most significant teleological cause. [Mid/Bot] Feature importance attributions for the slice before and during/after the queried subsequence correctly rank mechanistic causes. Violin plots show 5-fold cross-validation repeated 7 times.
• Figure 5: Changes to causal attributions with [Top] different sample sizes and [Bot] different smoothing weights for present-future mechanistic causes in conversation S1-A. Shaded regions are bootstrapped 95% confidence intervals.