Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Importance Sampling for Statistical Certification of Viable Initial Sets

Elizabeth Dietrich, Hanna Krasowski, Vegard Flovik, Murat Arcak

Abstract

We study the problem of statistically certifying viable initial sets (VISs) -- sets of initial conditions whose trajectories satisfy a given control specification. While VISs can be obtained from model-based methods, these methods typically rely on simplified models. We propose a simulation-based framework to certify VISs by estimating the probability of specification violations under a high-fidelity or black-box model. Since detecting these violations may be challenging due to their scarcity, we propose a sample-efficient framework that leverages importance sampling to target high-risk regions. We derive an empirical Bernstein inequality for weighted random variables, enabling finite-sample guarantees for importance sampling estimators. We demonstrate the effectiveness of the proposed approach on two systems and show improved convergence of the resulting bounds on an Adaptive Cruise Control benchmark.

Importance Sampling for Statistical Certification of Viable Initial Sets

Abstract

We study the problem of statistically certifying viable initial sets (VISs) -- sets of initial conditions whose trajectories satisfy a given control specification. While VISs can be obtained from model-based methods, these methods typically rely on simplified models. We propose a simulation-based framework to certify VISs by estimating the probability of specification violations under a high-fidelity or black-box model. Since detecting these violations may be challenging due to their scarcity, we propose a sample-efficient framework that leverages importance sampling to target high-risk regions. We derive an empirical Bernstein inequality for weighted random variables, enabling finite-sample guarantees for importance sampling estimators. We demonstrate the effectiveness of the proposed approach on two systems and show improved convergence of the resulting bounds on an Adaptive Cruise Control benchmark.

Paper Structure

This paper contains 16 sections, 3 theorems, 24 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Empirical Bernstein Bound
  5. Binomial Tail Bound
  6. Importance Sampling (IS)
  7. Problem Statement
  8. Methodology
  9. Adaptive Exploration and Importance Sampling
  10. Estimator and Certification
  11. Failure Characterization
  12. Empirical Results
  13. Failure Discovery with Gaussian Processes
  14. Adaptive Cruise Control
  15. Quadrotor Flight Scenario
  16. ...and 1 more sections

Key Result

Theorem 1

Let ${X},X_1, \dots, X_N$ be i.i.d. random variables with values in $[0, 1]$ and let $\beta > 0$. Then, with probability at least $1-\beta$, where $\hat{V}_{X}$ is the sample variance, $\blacktriangleleft$$\blacktriangleleft$

Figures (4)

  • Figure E1: ACC candidate VIS $\mathcal{C}$ (blue) and learned failure-prone set $\mathcal{F}$ (red) with zoom in of boundary region. The over-approximative $\mathcal{F}$ does not capture all failure points ($\times$), due to uncertainty in the GP estimate.
  • Figure E2: Top: ACC $\epsilon$ convergence to true failure probability. Bottom: Convergence rate of ACC estimators. The IS estimator converges rapidly to the true failure probability, closely following a $-1.0$ slope, whereas the binomial estimator converges with more variance at a rate close to $-0.5$. In this case study, $\alpha$ has a negligible effect on the IS estimator.
  • Figure E3: Quadrotor GP surrogate model with failures and the learned failure set $\mathcal{F}$. Failures are concentrated at low altitudes near the learned boundary.
  • Figure E4: Nominal, surrogate, and proposed distributions: the defensive mixture increases the probability of sampling from the failure region.

Theorems & Definitions (5)

  • Theorem 1: Theorem 4, maurer2009empirical
  • Definition 1: Binomial Tail Inversion
  • Definition 2: Importance-Weighted Loss
  • Lemma 1: Expected Value of ${Z}$
  • Theorem 2: Importance-Weighted PAC Guarantee