Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space

Gianmarco De Vita, Nargiz Humbatova, Paolo Tonella

Abstract

Testing Deep Learning (DL)-based systems is an open challenge. Although it is relatively easy to find inputs that cause a DL model to misbehave, the grouping of inputs by features that make the DL model under test fail is largely unexplored. Existing approaches for DL testing introduce perturbations that may focus on specific failure-inducing features, while neglecting others that belong to different regions of the feature space. In this paper, we create an explicit topographical map of the input feature space. Our approach, named TopoMap, is both black-box and model-agnostic as it relies solely on features that characterise the input space. To discriminate the inputs according to the specific features they share, we first apply dimensionality reduction to obtain input embeddings, which are then subjected to clustering. Each DL model might require specific embedding computations and clustering algorithms to achieve a meaningful separation of inputs into discriminative groups. We propose a novel way to evaluate alternative configurations of embedding and clustering techniques. We used a deep neural network (DNN) as an approximation of a human evaluator who could tell whether a pair of clusters can be discriminated based on the features of the included elements. We use such a DNN to automatically select the optimal topographical map of the inputs among all those that are produced by different embedding/clustering configurations. The evaluation results show that the maps generated by TopoMap consist of distinguishable and meaningful regions. In addition, we evaluate the effectiveness of TopoMap using mutation analysis. In particular, we assess whether the clusters in our topographical map allow for an effective selection of mutation-killing inputs. Experimental results show that our approach outperforms random selection by 35% on average on killable mutants; by 61% on non-killable ones.

TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space

Abstract

Testing Deep Learning (DL)-based systems is an open challenge. Although it is relatively easy to find inputs that cause a DL model to misbehave, the grouping of inputs by features that make the DL model under test fail is largely unexplored. Existing approaches for DL testing introduce perturbations that may focus on specific failure-inducing features, while neglecting others that belong to different regions of the feature space. In this paper, we create an explicit topographical map of the input feature space. Our approach, named TopoMap, is both black-box and model-agnostic as it relies solely on features that characterise the input space. To discriminate the inputs according to the specific features they share, we first apply dimensionality reduction to obtain input embeddings, which are then subjected to clustering. Each DL model might require specific embedding computations and clustering algorithms to achieve a meaningful separation of inputs into discriminative groups. We propose a novel way to evaluate alternative configurations of embedding and clustering techniques. We used a deep neural network (DNN) as an approximation of a human evaluator who could tell whether a pair of clusters can be discriminated based on the features of the included elements. We use such a DNN to automatically select the optimal topographical map of the inputs among all those that are produced by different embedding/clustering configurations. The evaluation results show that the maps generated by TopoMap consist of distinguishable and meaningful regions. In addition, we evaluate the effectiveness of TopoMap using mutation analysis. In particular, we assess whether the clusters in our topographical map allow for an effective selection of mutation-killing inputs. Experimental results show that our approach outperforms random selection by 35% on average on killable mutants; by 61% on non-killable ones.

Paper Structure

This paper contains 22 sections, 10 equations, 8 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Black-box Testing of DNNs
  4. Input Prioritisation
  5. Input Partition Testing
  6. DL Faults
  7. DeepHyperion
  8. Background
  9. Embedding Computation
  10. Clustering
  11. Approach
  12. Automated Selection of the Number of Clusters $k$
  13. Automated Assessment of the Quality of Clusters
  14. Experimental Procedure
  15. Subjects
  16. ...and 7 more sections

Figures (8)

  • Figure 1: Topographical map of aggregations of test input clusters killing various mutants (Udacity steering-angle prediction dataset)
  • Figure 2: Cluster generation and evaluation pipeline
  • Figure 3: Computation of the weighted pairwise accuracy on a three-cluster prediction example
  • Figure 4: Empirical design of the human evaluation study
  • Figure 5: Structure of the mutation-based evaluation pipeline
  • ...and 3 more figures