Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ACAV: A Framework for Automatic Causality Analysis in Autonomous Vehicle Accident Recordings

Huijia Sun, Christopher M. Poskitt, Yang Sun, Jun Sun, Yuqi Chen

TL;DR

ACAV tackles the lack of causal insight in autonomous-vehicle accident recordings by introducing a two-stage framework that first reduces recordings through feature-based segmentation and weighted voting, and then performs causality analysis with safety specifications and the Causality Analysis Tool (CAT) on Station-Time graphs. The approach yields high rates of causal-event identification across both accident recordings and fault-injected data, while significantly reducing the amount of data engineers must examine. It demonstrates strong results on the Apollo platform with SVL, identifying multiple causal event types and achieving high precision and recall in fault scenarios, and it shows promising generalizability to other ADSs. This work offers a practical, automatable path to diagnosing root causes in AV accidents, enabling targeted improvements in perception, prediction, and planning modules and informing future repair-oriented research.

Abstract

The rapid progress of autonomous vehicles~(AVs) has brought the prospect of a driverless future closer than ever. Recent fatalities, however, have emphasized the importance of safety validation through large-scale testing. Multiple approaches achieve this fully automatically using high-fidelity simulators, i.e., by generating diverse driving scenarios and evaluating autonomous driving systems~(ADSs) against different test oracles. While effective at finding violations, these approaches do not identify the decisions and actions that \emph{caused} them -- information that is critical for improving the safety of ADSs. To address this challenge, we propose ACAV, an automated framework designed to conduct causality analysis for AV accident recordings in two stages. First, we apply feature extraction schemas based on the messages exchanged between ADS modules, and use a weighted voting method to discard frames of the recording unrelated to the accident. Second, we use safety specifications to identify safety-critical frames and deduce causal events by applying CAT -- our causal analysis tool -- to a station-time graph. We evaluate ACAV on the Apollo ADS, finding that it can identify five distinct types of causal events in 93.64% of 110 accident recordings generated by an AV testing engine. We further evaluated ACAV on 1206 accident recordings collected from versions of Apollo injected with specific faults, finding that it can correctly identify causal events in 96.44% of the accidents triggered by prediction errors, and 85.73% of the accidents triggered by planning errors.

ACAV: A Framework for Automatic Causality Analysis in Autonomous Vehicle Accident Recordings

TL;DR

ACAV tackles the lack of causal insight in autonomous-vehicle accident recordings by introducing a two-stage framework that first reduces recordings through feature-based segmentation and weighted voting, and then performs causality analysis with safety specifications and the Causality Analysis Tool (CAT) on Station-Time graphs. The approach yields high rates of causal-event identification across both accident recordings and fault-injected data, while significantly reducing the amount of data engineers must examine. It demonstrates strong results on the Apollo platform with SVL, identifying multiple causal event types and achieving high precision and recall in fault scenarios, and it shows promising generalizability to other ADSs. This work offers a practical, automatable path to diagnosing root causes in AV accidents, enabling targeted improvements in perception, prediction, and planning modules and informing future repair-oriented research.

Abstract

The rapid progress of autonomous vehicles~(AVs) has brought the prospect of a driverless future closer than ever. Recent fatalities, however, have emphasized the importance of safety validation through large-scale testing. Multiple approaches achieve this fully automatically using high-fidelity simulators, i.e., by generating diverse driving scenarios and evaluating autonomous driving systems~(ADSs) against different test oracles. While effective at finding violations, these approaches do not identify the decisions and actions that \emph{caused} them -- information that is critical for improving the safety of ADSs. To address this challenge, we propose ACAV, an automated framework designed to conduct causality analysis for AV accident recordings in two stages. First, we apply feature extraction schemas based on the messages exchanged between ADS modules, and use a weighted voting method to discard frames of the recording unrelated to the accident. Second, we use safety specifications to identify safety-critical frames and deduce causal events by applying CAT -- our causal analysis tool -- to a station-time graph. We evaluate ACAV on the Apollo ADS, finding that it can identify five distinct types of causal events in 93.64% of 110 accident recordings generated by an AV testing engine. We further evaluated ACAV on 1206 accident recordings collected from versions of Apollo injected with specific faults, finding that it can correctly identify causal events in 96.44% of the accidents triggered by prediction errors, and 85.73% of the accidents triggered by planning errors.
Paper Structure (17 sections, 1 equation, 6 figures, 9 tables, 1 algorithm)

This paper contains 17 sections, 1 equation, 6 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Example
  3. Multi-Module ADSs
  4. Motivating Example
  5. Framework Design
  6. Stage #1: Accident Recording Simplification
  7. Stage #2: Causality Analysis
  8. Evaluation
  9. Research Questions & Evaluation Metrics
  10. Experiments and Discussion
  11. RQ1
  12. RQ2
  13. RQ3
  14. RQ4
  15. Threats to Validity
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Overview of $\mathtt{ACAV}$: the first stage vectorizes data exchanged between ADS modules and discards recording segments irrelevant to the accident; the second stage performs a causality analysis using the $\mathtt{CAT}$ tool
  • Figure 2: A visual example of an ST graph
  • Figure 3: Motivating example: six key scenes from a recording of an AV accident
  • Figure 4: Speed planning based on an ST graph
  • Figure 5: $\mathtt{CAT}$ for causality analysis
  • ...and 1 more figures