Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Querying Labeled Time Series Data with Scenario Programs

Edward Kim, Devan Shanker, Varun Bharadwaj, Hongbeen Park, Jinkyu Kim, Hazem Torfah, Daniel J Fremont, Sanjit A Seshia

TL;DR

A formal definition of how labeled time series sensor data can match an abstract scenario, represented as a scenario program using the Scenic probabilistic programming language is introduced, and a querying algorithm is presented that, given a scenario program and a labeled dataset, identifies the subset of data that matches the specified scenario.

Abstract

Simulation-based testing has become a crucial complement to road testing for ensuring the safety of cyber physical systems (CPS). As a result, significant research efforts have been directed toward identifying failure scenarios within simulation environments. However, a critical question remains. Are the AV failure scenarios discovered in simulation reproducible on actual systems in the real world? The sim-to-real gap caused by differences between simulated and real sensor data means that failure scenarios identified in simulation might either be artifacts of synthetic sensor data or actual issues that also occur with real sensor data. To address this, an effective approach to validating simulated failure scenarios is to locate occurrences of these scenarios within real-world datasets and verify whether the failure persists on the datasets. To this end, we introduce a formal definition of how labeled time series sensor data can match an abstract scenario, represented as a scenario program using the Scenic probabilistic programming language. We present a querying algorithm that, given a scenario program and a labeled dataset, identifies the subset of data that matches the specified scenario. Our experiment shows that our algorithm is more accurate and orders of magnitude faster in querying scenarios than the state-of-the-art commercial vision large language models, and can scale with the duration of queried time series data.

Querying Labeled Time Series Data with Scenario Programs

TL;DR

A formal definition of how labeled time series sensor data can match an abstract scenario, represented as a scenario program using the Scenic probabilistic programming language is introduced, and a querying algorithm is presented that, given a scenario program and a labeled dataset, identifies the subset of data that matches the specified scenario.

Abstract

Simulation-based testing has become a crucial complement to road testing for ensuring the safety of cyber physical systems (CPS). As a result, significant research efforts have been directed toward identifying failure scenarios within simulation environments. However, a critical question remains. Are the AV failure scenarios discovered in simulation reproducible on actual systems in the real world? The sim-to-real gap caused by differences between simulated and real sensor data means that failure scenarios identified in simulation might either be artifacts of synthetic sensor data or actual issues that also occur with real sensor data. To address this, an effective approach to validating simulated failure scenarios is to locate occurrences of these scenarios within real-world datasets and verify whether the failure persists on the datasets. To this end, we introduce a formal definition of how labeled time series sensor data can match an abstract scenario, represented as a scenario program using the Scenic probabilistic programming language. We present a querying algorithm that, given a scenario program and a labeled dataset, identifies the subset of data that matches the specified scenario. Our experiment shows that our algorithm is more accurate and orders of magnitude faster in querying scenarios than the state-of-the-art commercial vision large language models, and can scale with the duration of queried time series data.

Paper Structure

This paper contains 22 sections, 1 theorem, 10 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Overview
  3. Problem Formulation
  4. Methodology
  5. Supported Scenic Fragment
  6. Input-output hierarchical finite state machines
  7. Syntax-Directed Translation to HFSMs
  8. Query Algorithm
  9. Correctness of the Query Algorithm
  10. An Example Query
  11. Experiment
  12. Accuracy Experiment
  13. Scalability Experiment
  14. Conclusion
  15. Proof of Theorem 1
  16. ...and 7 more sections

Key Result

theorem thmcountertheorem

Given a Scenic program, a label trace, and an integer $m\in\mathbb{N}$, our algorithm outputs True if and only if the label trace matches the program for a window of length $m$; otherwise, the algorithm outputs False.

Figures (10)

  • Figure 1: A Scenic program modeling an ego car making a lane change to avoid a car in front.
  • Figure 2: Translation of Scenic syntax fragment to hierarchical FSM. The green boxes abstractly represent finite state machines, while the yellow boxes are states. The state with the black dot represents a termination state. The variables with $T$ subscript are Boolean predicates which indicates whether the corresponding machine terminated.
  • Figure 3: A hierarchical finite state machine representing the Scenic program in Fig. \ref{['fig:scenic-program']}. The guard, $g$, represents the interrupt condition, i.e. (distance from ego to otherCar) < Range(1,15) meters.
  • Figure 4: A snapshot from a matching video per scenario from Scenario 1 through 4 (clockwise from left top).
  • Figure 5: Scalability experiments of query runtime with respect to the increasing duration of label traces (left) and the number of objects in the program (right)
  • ...and 5 more figures

Theorems & Definitions (1)

  • theorem thmcountertheorem