Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STADA: Specification-based Testing for Autonomous Driving Agents

Joy Saha, Trey Woodlief, Sebastian Elbaum, Matthew B. Dwyer

TL;DR

STADA is a Specification-based Test generation framework for Autonomous Driving Agents that systematically generates the space of scenarios defined by a formal specification expressed in temporal logic (LTLf) and matches the coverage of the best baseline with 6 times fewer simulations.

Abstract

Simulation-based testing has become a standard approach to validating autonomous driving agents prior to real-world deployment. A high-quality validation campaign will exercise an agent in diverse contexts comprised of varying static environments, e.g., lanes, intersections, signage, and dynamic elements, e.g., vehicles and pedestrians. To achieve this, existing test generation techniques rely on template-based, manually constructed, or random scenario generation. When applied to validate formally specified safety requirements, such methods either require significant human effort or run the risk of missing important behavior related to the requirement. To address this gap, we present STADA, a Specification-based Test generation framework for Autonomous Driving Agents that systematically generates the space of scenarios defined by a formal specification expressed in temporal logic (LTLf). Given a specification, STADA constructs all distinct initial scenes, a diverse space of continuations of those scenes, and simulations that reflect the behaviors of the specification. Evaluation of STADA on a variety of LTLf specifications formalized in SCENEFLOW using three complementary coverage criteria demonstrates that STADA yields more than 2x higher coverage than the best baseline on the finest criteria and a 75% increase for the coarsest criteria. Moreover, it matches the coverage of the best baseline with 6 times fewer simulations. While set in the context of autonomous driving, the approach is applicable to other domains with rich simulation environments.

STADA: Specification-based Testing for Autonomous Driving Agents

TL;DR

STADA is a Specification-based Test generation framework for Autonomous Driving Agents that systematically generates the space of scenarios defined by a formal specification expressed in temporal logic (LTLf) and matches the coverage of the best baseline with 6 times fewer simulations.

Abstract

Simulation-based testing has become a standard approach to validating autonomous driving agents prior to real-world deployment. A high-quality validation campaign will exercise an agent in diverse contexts comprised of varying static environments, e.g., lanes, intersections, signage, and dynamic elements, e.g., vehicles and pedestrians. To achieve this, existing test generation techniques rely on template-based, manually constructed, or random scenario generation. When applied to validate formally specified safety requirements, such methods either require significant human effort or run the risk of missing important behavior related to the requirement. To address this gap, we present STADA, a Specification-based Test generation framework for Autonomous Driving Agents that systematically generates the space of scenarios defined by a formal specification expressed in temporal logic (LTLf). Given a specification, STADA constructs all distinct initial scenes, a diverse space of continuations of those scenes, and simulations that reflect the behaviors of the specification. Evaluation of STADA on a variety of LTLf specifications formalized in SCENEFLOW using three complementary coverage criteria demonstrates that STADA yields more than 2x higher coverage than the best baseline on the finest criteria and a 75% increase for the coarsest criteria. Moreover, it matches the coverage of the best baseline with 6 times fewer simulations. While set in the context of autonomous driving, the approach is applicable to other domains with rich simulation environments.
Paper Structure (16 sections, 3 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 16 sections, 3 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work and Background
  3. Coverage and Test Generation for AVs
  4. Specifying AV Properties
  5. Approach
  6. Key Insight
  7. Approach Overview
  8. RG Generation
  9. Initial Scene and Path Generation
  10. Simulation
  11. Evaluation
  12. Design
  13. Treatments
  14. RQ1: Coverage Results
  15. RQ2: Efficiency
  16. ...and 1 more sections

Figures (3)

  • Figure 1: (a) Relational graph for the precondition of $\varphi$ shown in blue. (b) The ego (red) and NPC (black) paths. (c) Initial scene satisfies $behind\_bike$ and $bike\_safe\_distance$ ($in\_front\_bike$ pending). (d) During passing, $bike\_safe\_distance$ is violated. (e) After passing, $in\_front\_bike$ is satisfied. The NPC bike is marked with a red arrow and the ego is a blue car.
  • Figure 2: Overview of STADA.
  • Figure 3: Mean coverage ($cov_1$) vs. number of simulations across different treatments.