Mitigating Side Effects in Multi-Agent Systems Using Blame Assignment

TL;DR

This work model the problem of mitigating NSEs in a cooperative multi-agent system as a bi-objective lexicographic decentralized Markov decision process, and decomposed into individual penalties for each robot using credit assignment, which facilitates decentralized policy computation.

Abstract

When independently trained or designed robots are deployed in a shared environment, their combined actions can lead to unintended negative side effects (NSEs). To ensure safe and efficient operation, robots must optimize task performance while minimizing the penalties associated with NSEs, balancing individual objectives with collective impact. We model the problem of mitigating NSEs in a cooperative multi-agent system as a bi-objective lexicographic decentralized Markov decision process. We assume independence of transitions and rewards with respect to the robots' tasks, but the joint NSE penalty creates a form of dependence in this setting. To improve scalability, the joint NSE penalty is decomposed into individual penalties for each robot using credit assignment, which facilitates decentralized policy computation. We empirically demonstrate, using mobile robots and in simulation, the effectiveness and scalability of our approach in mitigating NSEs.

Figures (5)

• Figure 1: Comparison of the paths taken by two TurtleBots performing delivery tasks in our indoor setup. The robots, initially unaware about the side effects of their actions, receive a joint penalty when one or both are in NSE states marked by X. The robots must update their behavior to complete tasks while mitigating NSEs.
• Figure 2: Overview of our solution approach. Agents independently compute policies to complete tasks described by $R_1^i$ (Naive policy). The NSE Monitor computes the NSE penalty for the joint policy $\vec{\pi}$. The Blame Resolver assigns a blame value for each agent, by evaluating counterfactual scenarios specific to each agent, as illustrated with warehouse robots handling different-sized boxes. Individual penalty functions $R^i_N$ are derived for each agent, based on the estimate blame. Agents then recompute their policies by solving the bi-objective problem with $R_1^i \succ R_N^i$, where $\succ$ denotes preference ordering over the objectives and their associated reward functions.
• Figure 3: Instances of environments from (a) salp, (b) overcooked, and (c) warehouse domains used in our experiments.
• Figure 4: Average NSE penalty and standard deviation for varying % of agents undergoing policy update in each domain with $25$ agents.
• Figure 5: Average NSE penalties and standard deviations, averaged over five problem instances with varying number of agents.

Theorems & Definitions (5)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5