Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-Shot Scenario Testing for Autonomous Vehicles Based on Neighborhood Coverage and Similarity

Shu Li, Jingxuan Yang, Honglin He, Yi Zhang, Jianming Hu, Shuo Feng

TL;DR

This paper forms for the first time the "few-shot testing" (FST) problem and proposes a systematic framework to address this challenge and shows a notable reduction in evaluation error and variance of the method compared to conventional testing methods.

Abstract

Testing and evaluating the safety performance of autonomous vehicles (AVs) is essential before the large-scale deployment. Practically, the number of testing scenarios permissible for a specific AV is severely limited by tight constraints on testing budgets and time. With the restrictions imposed by strictly restricted numbers of tests, existing testing methods often lead to significant uncertainty or difficulty to quantifying evaluation results. In this paper, we formulate this problem for the first time the "few-shot testing" (FST) problem and propose a systematic framework to address this challenge. To alleviate the considerable uncertainty inherent in a small testing scenario set, we frame the FST problem as an optimization problem and search for the testing scenario set based on neighborhood coverage and similarity. Specifically, under the guidance of better generalization ability of the testing scenario set on AVs, we dynamically adjust this set and the contribution of each testing scenario to the evaluation result based on coverage, leveraging the prior information of surrogate models (SMs). With certain hypotheses on SMs, a theoretical upper bound of evaluation error is established to verify the sufficiency of evaluation accuracy within the given limited number of tests. The experiment results on cut-in scenarios demonstrate a notable reduction in evaluation error and variance of our method compared to conventional testing methods, especially for situations with a strict limit on the number of scenarios.

Few-Shot Scenario Testing for Autonomous Vehicles Based on Neighborhood Coverage and Similarity

TL;DR

This paper forms for the first time the "few-shot testing" (FST) problem and proposes a systematic framework to address this challenge and shows a notable reduction in evaluation error and variance of the method compared to conventional testing methods.

Abstract

Testing and evaluating the safety performance of autonomous vehicles (AVs) is essential before the large-scale deployment. Practically, the number of testing scenarios permissible for a specific AV is severely limited by tight constraints on testing budgets and time. With the restrictions imposed by strictly restricted numbers of tests, existing testing methods often lead to significant uncertainty or difficulty to quantifying evaluation results. In this paper, we formulate this problem for the first time the "few-shot testing" (FST) problem and propose a systematic framework to address this challenge. To alleviate the considerable uncertainty inherent in a small testing scenario set, we frame the FST problem as an optimization problem and search for the testing scenario set based on neighborhood coverage and similarity. Specifically, under the guidance of better generalization ability of the testing scenario set on AVs, we dynamically adjust this set and the contribution of each testing scenario to the evaluation result based on coverage, leveraging the prior information of surrogate models (SMs). With certain hypotheses on SMs, a theoretical upper bound of evaluation error is established to verify the sufficiency of evaluation accuracy within the given limited number of tests. The experiment results on cut-in scenarios demonstrate a notable reduction in evaluation error and variance of our method compared to conventional testing methods, especially for situations with a strict limit on the number of scenarios.
Paper Structure (14 sections, 22 equations, 4 figures, 1 table)

This paper contains 14 sections, 22 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Performance Index Quantification with NDE
  4. Statistical Testing Problem
  5. Few-Shot Testing Problem
  6. Few-Shot Testing Framework
  7. Upper Bound of Estimation Error
  8. Coverage and Similarity-Based Estimation Strategy
  9. Optimizing for a Few-Shot Scenario Set
  10. Case Study
  11. Cut-in Scenario
  12. Experiment Settings
  13. Evaluation Results
  14. Conclusion

Figures (4)

  • Figure 1: Cut-in scenario.
  • Figure 2: Illustration of basic testing methods and SMs. Scenarios with smaller range and range rate in the left side of boundary would encounter crashes in simulation. The saturability of background demonstrate the exposure frequency in NDD.
  • Figure 3: Example of 20 samples and the coverage given by FST method.
  • Figure 4: Testing result of $n = 5, 10, 20$ samples with different methods repeated in 100 experiments.