Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling

TL;DR

Bayesian adaptive multifidelity sampling (BAMS) is introduced, which leverages the power of adaptive Bayesian sampling to achieve efficient discovery while simultaneously estimating the rate of adverse events.

Abstract

Ensuring the safety of autonomous vehicles (AVs) requires both accurate estimation of their performance and efficient discovery of potential failure cases. This paper introduces Bayesian adaptive multifidelity sampling (BAMS), which leverages the power of adaptive Bayesian sampling to achieve efficient discovery while simultaneously estimating the rate of adverse events. BAMS prioritizes exploration of regions with potentially low performance, leading to the identification of novel and critical scenarios that traditional methods might miss. Using real-world AV data we demonstrate that BAMS discovers 10 times as many issues as Monte Carlo (MC) and importance sampling (IS) baselines, while at the same time generating rate estimates with variances 15 and 6 times narrower than MC and IS baselines respectively.

Figures (4)

• Figure 1: Illustration of BAMS. a) Iterative loop of our approach, as defined in Algorithm \ref{['alg:bams']} (Appendix \ref{['sec:alg']}). In each iteration, we separate the inputs into clusters, solve problem \ref{['eq:opt-prob-multi']} over each cluster, and then select the final points from the candidates in all the clusters. After performing simulations over these points, we update the GP posterior. For simplicity, we neglect illustrating multiple fidelities in this graphic. b) Distribution of samples $P$ for a simple synthetic setting. Samples are colored red if $f(x) \le \gamma$ and blue otherwise. c) Points selected over three iterations (batches) of our algorithm in the synthetic setting. Initial exploration over the state space in the first iteration transitions to targeted uncertainty reduction around the regions of interest in the third iteration.
• Figure 2: Retention-recall curves for AV experiments. The $x$-axis represents the retention budget scaled by $p_{\gamma} N$ while the $y$-axis shows the recall at each corresponding retention budget.
• Figure 3: We select five samples from Batch 3 data using a DPP sampler to maximize diversity. The ego vehicle is blue and the snapshot corresponds to the timestamp at which the minimum TTC occurs; large transparent circles indicate the trajectory history, and small circles represent the future trajectory. The BAMS samples showcase busy intersections, pedestrians, and cyclists, whereas MCM-GP samples have less diversity.
• Figure 4: AV setting $90\%$ confidence intervals for IS rate estimate with $K=5p_{\gamma}N$ samples. The ground truth is $p_{\gamma}=0.01$. Smaller error bars indicate a more precise estimator.

Theorems & Definitions (2)

• Proposition 1
• Corollary 1