Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guiding the Search Towards Failure-Inducing Test Inputs Using Support Vector Machines

Lev Sorokin, Niklas Kerscher

TL;DR

The paper addresses the challenge of efficiently finding failure revealing inputs in large SBST search spaces. It introduces NSGA-II-SVM, a surrogate guided by SVM classification to direct evolutionary search toward high risk regions, retraining after each iteration. In a preliminary AVP case study, NSGA-II-SVM outperforms random search and NSGA-II-DT, identifying about $34\%$ more distinct failure cases and achieving better quality indicators. This approach offers a practical, scalable means to accelerate the discovery of critical failure scenarios in safety-critical, learning-enabled systems and can be extended to other domains with large input spaces.

Abstract

In this paper, we present NSGA-II-SVM (Non-dominated Sorting Genetic Algorithm with Support Vector Machine Guidance), a novel learnable evolutionary and search-based testing algorithm that leverages Support Vector Machine (SVM) classification models to direct the search towards failure-revealing test inputs. Supported by genetic search, NSGA-II-SVM creates iteratively SVM-based models of the test input space, learning which regions in the search space are promising to be explored. A subsequent sampling and repetition of evolutionary search iterations allow to refine and make the model more accurate in the prediction. Our preliminary evaluation of NSGA-II-SVM by testing an Automated Valet Parking system shows that NSGA-II-SVM is more effective in identifying more critical test cases than a state of the art learnable evolutionary testing technique as well as naive random search.

Guiding the Search Towards Failure-Inducing Test Inputs Using Support Vector Machines

TL;DR

The paper addresses the challenge of efficiently finding failure revealing inputs in large SBST search spaces. It introduces NSGA-II-SVM, a surrogate guided by SVM classification to direct evolutionary search toward high risk regions, retraining after each iteration. In a preliminary AVP case study, NSGA-II-SVM outperforms random search and NSGA-II-DT, identifying about more distinct failure cases and achieving better quality indicators. This approach offers a practical, scalable means to accelerate the discovery of critical failure scenarios in safety-critical, learning-enabled systems and can be extended to other domains with large input spaces.

Abstract

In this paper, we present NSGA-II-SVM (Non-dominated Sorting Genetic Algorithm with Support Vector Machine Guidance), a novel learnable evolutionary and search-based testing algorithm that leverages Support Vector Machine (SVM) classification models to direct the search towards failure-revealing test inputs. Supported by genetic search, NSGA-II-SVM creates iteratively SVM-based models of the test input space, learning which regions in the search space are promising to be explored. A subsequent sampling and repetition of evolutionary search iterations allow to refine and make the model more accurate in the prediction. Our preliminary evaluation of NSGA-II-SVM by testing an Automated Valet Parking system shows that NSGA-II-SVM is more effective in identifying more critical test cases than a state of the art learnable evolutionary testing technique as well as naive random search.
Paper Structure (11 sections, 3 figures, 1 algorithm)

This paper contains 11 sections, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Search-based Testing Problem
  4. Support Vector Machines
  5. SVM Guided Testing
  6. Empirical Evaluation
  7. Experimental Setup
  8. Results
  9. Threats to Validity
  10. Related Work
  11. Discussion and Future Work

Figures (3)

  • Figure 1: Illustration of difference between boundary identification by a Decision Tree and Support Vector Machine for data from two classes (circle/triangle positive/negative data).
  • Figure 2: Overview of the main steps of NSGA-II-SVM. a) Genetic search using NSGA-II. b) Learned SVM-model for failing region prediction and sampling inside region. c) Genetic search with NSGA-II in subsequent iteration using evaluated samples and best solutions found. d) Refined SVM-model learning. Arrows indicate genetic operations.
  • Figure 3: Quality metric results averaged over 10 runs.