Generating Realistic, Diverse, and Fault-Revealing Inputs with Latent Space Interpolation for Testing Deep Neural Networks

TL;DR

This paper tackles the challenge of testing deep neural networks in safety-critical settings by proposing ARGUS, a black-box test-input generator that yields realistic, diverse, and fault-revealing samples. ARGUS achieves this by performing interpolation in a continuous latent space of a VQ-VAE and reconstructing samples through a dual-discriminator setup in both latent and input spaces, guided by a perturbation parameter λ. Empirical results on MNIST, CIFAR-10, and ImageNet with multiple architectures show ARGUS outperforms state-of-the-art baselines in realism, diversity, and fault exposure, with a 100% perturbation-success rate and up to 4x higher error rates; retraining with ARGUS data also improves accuracy. The work demonstrates practical benefits for black-box DNN testing and model improvement, including public release of the ARGUS implementation.

Abstract

Deep Neural Networks (DNNs) have been widely employed across various domains, including safety-critical systems, necessitating comprehensive testing to ensure their reliability. Although numerous DNN model testing methods have been proposed to generate adversarial samples that are capable of revealing faults, existing methods typically perturb samples in the input space and then mutate these based on feedback from the DNN model. These methods often result in test samples that are not realistic and with low-probability reveal faults. To address these limitations, we propose a black-box DNN test input generation method, ARGUS, to generate realistic, diverse, and fault-revealing test inputs. ARGUS first compresses samples into a continuous latent space and then perturbs the original samples by interpolating these with samples of different classes. Subsequently, we employ a vector quantizer and decoder to reconstruct adversarial samples back into the input space. Additionally, we employ discriminators both in the latent space and in the input space to ensure the realism of the generated samples. Evaluation of ARGUS in comparison with state-of-the-art black-box testing and white-box testing methods, shows that ARGUS excels in generating realistic and diverse adversarial samples relative to the target dataset, and ARGUS successfully perturbs all original samples and achieves up to 4 times higher error rate than the best baseline method. Furthermore, using these adversarial samples for model retraining can improve model classification accuracy.

Figures (3)

• Figure 1: VAE, VQ-VAE, and GAN Architecture
• Figure 2: Overall Framework of Argus
• Figure 3: Results of Manually Evaluating Sample Realism.