Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Targeted Deep Learning System Boundary Testing

Oliver Weißl, Amr Abdellatif, Xingcheng Chen, Giorgi Merabishvili, Vincenzo Riccio, Severin Kacianka, Andrea Stocco

TL;DR

Mimicry introduces a targeted boundary testing approach for DL classifiers by leveraging the disentangled latent space of style-based GANs to move inputs toward decision boundaries. It treats boundary discovery as a black-box optimization problem guided by SUT feedback, using a two-objective function for dynamic confidence balance and novelty in latent manipulations. Across five image datasets and a consistent WideResNet-50-2 backbone, Mimicry outperforms model-based and latent-space baselines (DeepJanus, Sinvad) in boundary proximity, input validity, and label-preservation, particularly as data complexity grows. The method demonstrates robust boundary generation with controllable, semantically meaningful changes, offering a practical pathway for evaluating DL reliability in real-world, high-dimensional input spaces.

Abstract

Evaluating the behavioral boundaries of deep learning (DL) systems is crucial for understanding their reliability across diverse, unseen inputs. Existing solutions fall short as they rely on untargeted random, model- or latent-based perturbations, due to difficulties in generating controlled input variations. In this work, we introduce Mimicry, a novel black-box test generator for fine-grained, targeted exploration of DL system boundaries. Mimicry performs boundary testing by leveraging the probabilistic nature of DL outputs to identify promising directions for exploration. It uses style-based GANs to disentangle input representations into content and style components, enabling controlled feature mixing to approximate the decision boundary. We evaluated Mimicry's effectiveness in generating boundary inputs for five widely used DL image classification systems of increasing complexity, comparing it to two baseline approaches. Our results show that Mimicry consistently identifies inputs closer to the decision boundary. It generates semantically meaningful boundary test cases that reveal new functional (mis)behaviors, while the baselines produce mainly corrupted or invalid inputs. Thanks to its enhanced control over latent space manipulations, Mimicry remains effective as dataset complexity increases, maintaining competitive diversity and higher validity rates, confirmed by human assessors.

Targeted Deep Learning System Boundary Testing

TL;DR

Mimicry introduces a targeted boundary testing approach for DL classifiers by leveraging the disentangled latent space of style-based GANs to move inputs toward decision boundaries. It treats boundary discovery as a black-box optimization problem guided by SUT feedback, using a two-objective function for dynamic confidence balance and novelty in latent manipulations. Across five image datasets and a consistent WideResNet-50-2 backbone, Mimicry outperforms model-based and latent-space baselines (DeepJanus, Sinvad) in boundary proximity, input validity, and label-preservation, particularly as data complexity grows. The method demonstrates robust boundary generation with controllable, semantically meaningful changes, offering a practical pathway for evaluating DL reliability in real-world, high-dimensional input spaces.

Abstract

Evaluating the behavioral boundaries of deep learning (DL) systems is crucial for understanding their reliability across diverse, unseen inputs. Existing solutions fall short as they rely on untargeted random, model- or latent-based perturbations, due to difficulties in generating controlled input variations. In this work, we introduce Mimicry, a novel black-box test generator for fine-grained, targeted exploration of DL system boundaries. Mimicry performs boundary testing by leveraging the probabilistic nature of DL outputs to identify promising directions for exploration. It uses style-based GANs to disentangle input representations into content and style components, enabling controlled feature mixing to approximate the decision boundary. We evaluated Mimicry's effectiveness in generating boundary inputs for five widely used DL image classification systems of increasing complexity, comparing it to two baseline approaches. Our results show that Mimicry consistently identifies inputs closer to the decision boundary. It generates semantically meaningful boundary test cases that reveal new functional (mis)behaviors, while the baselines produce mainly corrupted or invalid inputs. Thanks to its enhanced control over latent space manipulations, Mimicry remains effective as dataset complexity increases, maintaining competitive diversity and higher validity rates, confirmed by human assessors.
Paper Structure (35 sections, 10 equations, 14 figures, 4 tables)

This paper contains 35 sections, 10 equations, 14 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Testing Objectives for DL Systems
  4. Boundary Testing for DL Systems
  5. Style-Based Generative Adversarial Networks
  6. Methodology
  7. System under Test
  8. Manipulator
  9. Optimizer
  10. Objectives for Boundary Testing
  11. Empirical Study
  12. Research Questions
  13. Objects of Study
  14. Datasets
  15. System Under Test
  16. ...and 20 more sections

Figures (14)

  • Figure 1: Different types of boundary testing objectives for DL systems.
  • Figure 2: Mimicry component interactions./
  • Figure 3: Mixing features of an original image (blue car) with those of a target image (white truck) produces different outputs depending on the latent layers.
  • Figure 4: Latent manipulation approaches.
  • Figure 5: Boundary discovery between classes "5" and "6" in the MNIST benchmark.
  • ...and 9 more figures