Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Assessing Deep Learning Test Input Generators

Seif Mzoughi, Ahmed Haj yahmed, Mohamed Elshafei, Foutse Khomh, Diego Elias Costa

TL;DR

The paper tackles the challenge of evaluating robustness test input generators (TIGs) for deep learning by conducting a comprehensive empirical study of four state-of-the-art TIGs (DeepHunter, DeepFault, AdvGAN, SinVAD) across three model–dataset pairs (LeNet-5/MNIST, VGG16/CIFAR-10, EfficientNetB3/ImageNet-1K). It assesses robustness-revealing capability, naturalness, diversity, and efficiency using five metrics (DDR, ASR, LPIPS, PM, ET) and repeated trials to ensure statistical validity. Key findings reveal trade-offs: GAN-based TIGs perform best on simple data but struggle with high-resolution ImageNet; naturalness often inversely correlates with robustness; SinVAD provides high diversity on easy datasets but can produce invalid inputs on complex ones; and efficiency varies markedly across tools and datasets. The study offers practical guidance for TIG selection, discusses adaptation challenges for modern architectures, and provides a replication package to support ongoing robustness testing in real-world settings.

Abstract

Deep Learning (DL) systems are increasingly deployed in safety-critical applications, yet they remain vulnerable to robustness issues that can lead to significant failures. While numerous Test Input Generators (TIGs) have been developed to evaluate DL robustness, a comprehensive assessment of their effectiveness across different dimensions is still lacking. This paper presents a comprehensive assessment of four state-of-the-art TIGs--DeepHunter, DeepFault, AdvGAN, and SinVAD--across multiple critical aspects: fault-revealing capability, naturalness, diversity, and efficiency. Our empirical study leverages three pre-trained models (LeNet-5, VGG16, and EfficientNetB3) on datasets of varying complexity (MNIST, CIFAR-10, and ImageNet-1K) to evaluate TIG performance. Our findings reveal important trade-offs in robustness revealing capability, variation in test case generation, and computational efficiency across TIGs. The results also show that TIG performance varies significantly with dataset complexity, as tools that perform well on simpler datasets may struggle with more complex ones. In contrast, others maintain steadier performance or better scalability. This paper offers practical guidance for selecting appropriate TIGs aligned with specific objectives and dataset characteristics. Nonetheless, more work is needed to address TIG limitations and advance TIGs for real-world, safety-critical systems.

Towards Assessing Deep Learning Test Input Generators

TL;DR

The paper tackles the challenge of evaluating robustness test input generators (TIGs) for deep learning by conducting a comprehensive empirical study of four state-of-the-art TIGs (DeepHunter, DeepFault, AdvGAN, SinVAD) across three model–dataset pairs (LeNet-5/MNIST, VGG16/CIFAR-10, EfficientNetB3/ImageNet-1K). It assesses robustness-revealing capability, naturalness, diversity, and efficiency using five metrics (DDR, ASR, LPIPS, PM, ET) and repeated trials to ensure statistical validity. Key findings reveal trade-offs: GAN-based TIGs perform best on simple data but struggle with high-resolution ImageNet; naturalness often inversely correlates with robustness; SinVAD provides high diversity on easy datasets but can produce invalid inputs on complex ones; and efficiency varies markedly across tools and datasets. The study offers practical guidance for TIG selection, discusses adaptation challenges for modern architectures, and provides a replication package to support ongoing robustness testing in real-world settings.

Abstract

Deep Learning (DL) systems are increasingly deployed in safety-critical applications, yet they remain vulnerable to robustness issues that can lead to significant failures. While numerous Test Input Generators (TIGs) have been developed to evaluate DL robustness, a comprehensive assessment of their effectiveness across different dimensions is still lacking. This paper presents a comprehensive assessment of four state-of-the-art TIGs--DeepHunter, DeepFault, AdvGAN, and SinVAD--across multiple critical aspects: fault-revealing capability, naturalness, diversity, and efficiency. Our empirical study leverages three pre-trained models (LeNet-5, VGG16, and EfficientNetB3) on datasets of varying complexity (MNIST, CIFAR-10, and ImageNet-1K) to evaluate TIG performance. Our findings reveal important trade-offs in robustness revealing capability, variation in test case generation, and computational efficiency across TIGs. The results also show that TIG performance varies significantly with dataset complexity, as tools that perform well on simpler datasets may struggle with more complex ones. In contrast, others maintain steadier performance or better scalability. This paper offers practical guidance for selecting appropriate TIGs aligned with specific objectives and dataset characteristics. Nonetheless, more work is needed to address TIG limitations and advance TIGs for real-world, safety-critical systems.

Paper Structure

This paper contains 19 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Deep Learning Systems
  4. Deep Learning Testing
  5. Test Input Generators
  6. Study Design
  7. Models and Datasets
  8. DL Test Input Generators
  9. Generation of Synthetic Test Inputs
  10. Evaluation Criteria
  11. Results
  12. RQ1: Which TIG reveals more DNN robustness issues?
  13. RQ2: Which TIG generates more natural test cases?
  14. RQ3: Which TIG generates more diversified test cases?
  15. RQ4: Which TIG is more efficient in test case generation?
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Study Methodology for Assessing Test Input Generator (TIG) Performance
  • Figure 2: 3D Performance Comparison of DNN Testing Tools, illustrating Detection Rate (DDR), Perturbation Magnitude (PM), and Perceptual Similarity (LPIPS) metrics across increasing dataset complexity in ImageNet, CIFAR-10, and MNIST, highlighting each tool's effectiveness and robustness.
  • Figure 3: DDR Performance Trends of TIGs with Increasing Dataset Complexity, illustrating the scalability challenges encountered on more complex datasets.