Who is Responsible? Explaining Safety Violations in Multi-Agent Cyber-Physical Systems

TL;DR

An automated procedure to assign responsibility for safety violations to actions of any single agent in a principled manner is developed and the results indicate that the degree of responsibility (DoR) enhances explainability of decision-making and assigning accountability for actions of agents and their consequences.

Abstract

Multi-agent cyber-physical systems are present in a variety of applications. Agent decision-making can be affected due to errors induced by uncertain, dynamic operating environments or due to incorrect actions taken by an agent. When an erroneous decision that leads to a violation of safety is identified, assigning responsibility to individual agents is a key step toward preventing future accidents. Current approaches to carrying out such investigations require human labor or high degree of familiarity with operating environments. Automated strategies to assign responsibility can achieve a significant reduction in human effort and associated cognitive burden. In this paper, we develop an automated procedure to assign responsibility for safety violations to actions of any single agent in a principled manner. We base our approach on reasoning about safety violations in road safety. Given a safety violation, we use counterfactual reasoning to create alternative scenarios, showing how different outcomes could have occurred if certain actions had been replaced by others. We introduce the degree of responsibility (DoR) metric for each agent. The DoR, using the Shapley value, quantifies each agent's contribution to the safety violation, providing a basis to explain and justify decisions. We also develop heuristic techniques and methods based on agent interaction structures to improve scalability as agent numbers grow. We examine three safety violation cases from the National Highway Traffic Safety Administration (NHTSA). We run experiments using CARLA urban driving simulator. Results show the DoR improves the explainability of decisions and accountability for agent actions and their consequences.

Figures (4)

• Figure 1: This figure presents a road segment consisting of two lanes. The road segment is discretized into eight discrete locations. Four vehicles share the road segment.
• Figure 2: Scenario 1: This figure presents the initial locations of vehicles, the collision, and the discretized locations for Scenario 1. The red car is Agent 1 and the blue car is Agent 2. The initial locations of Agents 1 and 2 (labeled as A1 and A2) are presented in Fig. \ref{['fig:s1']}-(a). Fig. \ref{['fig:s1']}-(b) illustrates a collision, where Agent 1 collides with a pedestrian (P1). The road segment is discretized into 12 locations $\mathcal{L}=\{0,\ldots,11\}$ as shown in Fig. \ref{['fig:s1']}-(c). Red arrows represent paths observed in path $\rho$, and the green STOP sign represents the safe policy of Agent 1. Here the STOP sign indicates that the agent should take the stop action to avoid safety violation.
• Figure 3: Scenario 2: This figure presents the initial locations of vehicles, the collision, and the discretized locations for Scenario 2. The red car is Agent 1, the white SUV is Agent 2, and the green motorcycle is Agent 3. The initial locations of Agents 1, 2, and 3 (labeled as A1, A2, and A3) are presented in Fig. \ref{['fig:s2']}-(a). Fig. \ref{['fig:s2']}-(b) illustrates the collision, where Agent 1 collides with Agent 3. The road segment is discretized into 8 locations $\mathcal{L}=\{0,\ldots,7\}$, as shown in Fig. \ref{['fig:s2']}-(c). Red arrows represent the paths observed in path $\rho$, and the green STOP sign represents the safe policy of Agents 1 and 3. Here the STOP sign indicates that the agent should take the stop action to avoid safety violation.
• Figure 4: Scenario 3: This figure presents the initial locations of vehicles, the collision, and the discretized locations for Scenario 3. The white SUV is Agent 1, and the orange truck is Agent 2. The initial locations of Agents 1 and 2 (labeled as A1 and A2) are presented in Fig. \ref{['fig:s3']}-(a). Fig. \ref{['fig:s3']}-(b) illustrates the collision, where Agent 1 collides with Agent 2. The road segment is discretized into 9 locations $\mathcal{L}=\{0,\ldots,8\}$ as shown in Fig. \ref{['fig:s3']}-(c). Red arrows represent the paths observed in path $\rho$, and the green arrow represents the safe policy of Agent 1.

Theorems & Definitions (11)

• Definition 1: Multi-Agent Markov Decision Process (MMDP)
• Example 1
• Example 1: Continued
• Example 1: Continued
• Proposition 1
• Example 1: Continued
• Definition 2: $\epsilon$-Marginally Safe Policy
• Definition 3: $(c,\gamma)$ Exponential Decay Property qu2020scalable
• Lemma 1
• Proposition 2