DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

TL;DR

DeepGD addresses the challenge of testing deep neural networks under labeling cost by selecting small, informative test input subsets from large unlabeled pools. It casts test selection as a multi-objective NSGA-II optimization that jointly maximizes average uncertainty via the Gini score and input diversity via geometric diversity (GD), with GD computed from features extracted by a pre-trained VGG-16 network. A clustering-based fault estimation strategy (via features from mispredicted inputs and HDBSCAN) evaluates fault-revealing power through Fault Detection Rate (FDR), and the approach is validated on four image datasets and five DNNs against nine baselines. Results show DeepGD yields higher fault discovery, better retraining guidance (up to 38.03% optimization effectiveness on generated data), and more diverse test sets, with statistically significant improvements, albeit with longer compute times that remain practical when labeling costs dominate. These findings suggest DeepGD offers a practical, black-box solution to reduce labeling burden while enhancing DNN reliability and retraining outcomes.

Abstract

Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. Particularly, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness. In this paper, we propose DeepGD, a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high fault revealing power from large unlabeled datasets. DeepGD not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs. The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability: (1) White-box, coverage-based approaches fare poorly, (2) DeepGD outperforms existing black-box test selection approaches in terms of fault detection, and (3) DeepGD also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.

Figures (7)

• Figure 1: Illustration of the geometric diversity metric aghababaeyan2021black
• Figure 2: Examples of geometric diversity scores for two different subsets
• Figure 3: Relying on misprediction rates is misleading aghababaeyan2021black
• Figure 4: GD scores for subsets size=100
• Figure 5: GD scores for subsets size=300