Feature-Aware Test Generation for Deep Learning Models

TL;DR

Detect addresses the twin challenges of generalization and robustness in vision DL models by enabling feature-aware test generation in a disentangled latent space. It integrates StyleGAN-based perturbations in StyleSpace $\mathcal{S}$ with a Vision-Language Model for semantic attribution, separating task-relevant from task-irrelevant features and enabling targeted perturbations. The framework demonstrates improved decision-boundary discovery, exposes distinct shortcut behaviors across architectures, and supports model repair through lightweight fine-tuning. This approach provides interpretable, semantically grounded testing that informs debugging and robustness improvements in practical DL deployments.

Abstract

As deep learning models are widely used in software systems, test generation plays a crucial role in assessing the quality of such models before deployment. To date, the most advanced test generators rely on generative AI to synthesize inputs; however, these approaches remain limited in providing semantic insight into the causes of misbehaviours and in offering fine-grained semantic controllability over the generated inputs. In this paper, we introduce Detect, a feature-aware test generation framework for vision-based deep learning (DL) models that systematically generates inputs by perturbing disentangled semantic attributes within the latent space. Detect perturbs individual latent features in a controlled way and observes how these changes affect the model's output. Through this process, it identifies which features lead to behavior shifts and uses a vision-language model for semantic attribution. By distinguishing between task-relevant and irrelevant features, Detect applies feature-aware perturbations targeted for both generalization and robustness. Empirical results across image classification and detection tasks show that Detect generates high-quality test cases with fine-grained control, reveals distinct shortcut behaviors across model architectures (convolutional and transformer-based), and bugs that are not captured by accuracy metrics. Specifically, Detect outperforms a state-of-the-art test generator in decision boundary discovery and a leading spurious feature localization method in identifying robustness failures. Our findings show that fully fine-tuned convolutional models are prone to overfitting on localized cues, such as co-occurring visual traits, while weakly supervised transformers tend to rely on global features, such as environmental variances. These findings highlight the value of interpretable and feature-aware testing in improving DL model reliability.

Figures (11)

• Figure 1: System Architecture of Detect: It comprises two main components, a StyleGAN generator and the SUT, and operates across three domains: the latent $\mathcal{S}$-space, the image space, and the output space. It contains 1) a forward generation pass from an initial seed $z$ mapped into a set of style vectors $s$, which synthesize an image input passed to the SUT to obtain output logits; and 2) a backpropagation through SUT and synthesis network back to $s$.
• Figure 2: Example perturbations in StyleSpace for glasses classification. Each row shows a base image (left) and variants generated by perturbing a single feature (i.e., glasses, eyebrows, skin color). Titles: channel index, feature label, model output logit.
• Figure 3: Sequential Pipeline of Detect.
• Figure 4: Detect with Oracle Confidence and Misclassification.
• Figure 5: Example of Image Input to VLM.

Theorems & Definitions (3)

• Definition 1: Task-Relevant Feature
• Definition 2: Task-Irrelevant Feature
• Definition 3: Spurious Feature