Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

Kevin Kai-Chun Chang, Ekin Beyazit, Alberto Sangiovanni-Vincentelli, Tichakorn Wongpiromsarn, Sanjit A. Seshia

TL;DR

This work introduces ScenicRules, a benchmark for evaluating autonomous driving systems in stochastic environments under prioritized multi-objective specifications, and designs a Hierarchical Rulebook framework that encodes multiple objectives and their priority relations in an interpretable and adaptable manner.

Abstract

Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, obeying traffic rules, and making efficient progress. In many situations, these objectives cannot be satisfied simultaneously, and explicit priority relations naturally arise. Also, driving rules require context, so it is important to formally model the environment scenarios within which such rules apply. Existing benchmarks for evaluating autonomous vehicles lack such combinations of multi-objective prioritized rules and formal environment models. In this work, we introduce ScenicRules, a benchmark for evaluating autonomous driving systems in stochastic environments under prioritized multi-objective specifications. We first formalize a diverse set of objectives to serve as quantitative evaluation metrics. Next, we design a Hierarchical Rulebook framework that encodes multiple objectives and their priority relations in an interpretable and adaptable manner. We then construct a compact yet representative collection of scenarios spanning diverse driving contexts and near-accident situations, formally modeled in the Scenic language. Experimental results show that our formalized objectives and Hierarchical Rulebooks align well with human driving judgments and that our benchmark effectively exposes agent failures with respect to the prioritized objectives. Our benchmark can be accessed at https://github.com/BerkeleyLearnVerify/ScenicRules/.

ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

TL;DR

This work introduces ScenicRules, a benchmark for evaluating autonomous driving systems in stochastic environments under prioritized multi-objective specifications, and designs a Hierarchical Rulebook framework that encodes multiple objectives and their priority relations in an interpretable and adaptable manner.

Abstract

Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, obeying traffic rules, and making efficient progress. In many situations, these objectives cannot be satisfied simultaneously, and explicit priority relations naturally arise. Also, driving rules require context, so it is important to formally model the environment scenarios within which such rules apply. Existing benchmarks for evaluating autonomous vehicles lack such combinations of multi-objective prioritized rules and formal environment models. In this work, we introduce ScenicRules, a benchmark for evaluating autonomous driving systems in stochastic environments under prioritized multi-objective specifications. We first formalize a diverse set of objectives to serve as quantitative evaluation metrics. Next, we design a Hierarchical Rulebook framework that encodes multiple objectives and their priority relations in an interpretable and adaptable manner. We then construct a compact yet representative collection of scenarios spanning diverse driving contexts and near-accident situations, formally modeled in the Scenic language. Experimental results show that our formalized objectives and Hierarchical Rulebooks align well with human driving judgments and that our benchmark effectively exposes agent failures with respect to the prioritized objectives. Our benchmark can be accessed at https://github.com/BerkeleyLearnVerify/ScenicRules/.
Paper Structure (28 sections, 7 equations, 14 figures, 7 tables, 2 algorithms)

This paper contains 28 sections, 7 equations, 14 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Signal Temporal Logic (STL)
  5. Rulebook Specification
  6. Scenic Programming Language
  7. Our Benchmark
  8. Collection and Formalization of Objectives
  9. Hierarchical Rulebook Specifications
  10. Formalized Environment Scenarios in Scenic
  11. Common Scenarios
  12. Near-accident Scenarios
  13. Experiments
  14. RQ1: Evaluation of Hierarchical Rulebooks
  15. Performance and Generalizability
  16. ...and 13 more sections

Figures (14)

  • Figure 1: Real-world autonomous driving examples with multiple conflicting objectives.
  • Figure 2: An example of a Rulebook.
  • Figure 3: An example Scenic program.
  • Figure 4: Categorization of rules.
  • Figure 5: Our automated Scenic program generator.
  • ...and 9 more figures

Theorems & Definitions (10)

  • Definition 1: Rulebook
  • Definition 2: Error Value
  • Definition 3: Hierarchical Rulebook
  • Example 1: Emergency Vehicles
  • Example 2: Dense Traffic
  • Example 3: Degraded Performance
  • Example 4: Scenario Encoding
  • Definition 4: STL Syntax
  • Definition 5: STL Semantics
  • Definition 6: STL Robustness