FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection

TL;DR

FAST is a method that boosts existing prioritization methods through guided FeAture SelecTion, a method that quantifies the importance of each feature for the model’s correct predictions, and dynamically prunes the information from the noisy features during inference to derive a new probability vector for the uncertainty estimation.

Abstract

Due to the vast testing space, the increasing demand for effective and efficient testing of deep neural networks (DNNs) has led to the development of various DNN test case prioritization techniques. However, the fact that DNNs can deliver high-confidence predictions for incorrectly predicted examples, known as the over-confidence problem, causes these methods to fail to reveal high-confidence errors. To address this limitation, in this work, we propose FAST, a method that boosts existing prioritization methods through guided FeAture SelecTion. FAST is based on the insight that certain features may introduce noise that affects the model's output confidence, thereby contributing to high-confidence errors. It quantifies the importance of each feature for the model's correct predictions, and then dynamically prunes the information from the noisy features during inference to derive a new probability vector for the uncertainty estimation. With the help of FAST, the high-confidence errors and correctly classified examples become more distinguishable, resulting in higher APFD (Average Percentage of Fault Detection) values for test prioritization, and higher generalization ability for model enhancement. We conduct extensive experiments to evaluate FAST across a diverse set of model structures on multiple benchmark datasets to validate the effectiveness, efficiency, and scalability of FAST compared to the state-of-the-art prioritization techniques.

Figures (8)

• Figure 1: Illustration of the key limitation of existing uncertainty-based methods, which focus primarily on the test cases close to the decision boundary while neglecting the high-confidence errors. FAST helps to expose such high-confidence errors by dynamically suppressing their confidence, pulling them towards the low-confidence region.
• Figure 2: The feature contribution to the output confidence for the FASHION model (LeNet-5) and the CIFAR-10 model (VGG-16). The features are sorted in an increasing order based on their average contribution over the correctly classified examples with high confidence. The key facts are: 1) the most important features at the higher end significantly impact both correctly and incorrectly classified samples with high confidence, and 2) while the features at the lower end contribute the least to high-confidence correct classifications, they can still significantly affect high-confidence misclassifications.
• Figure 3: An overview of FAST framework. It selectively drops a set of noisy features during inference to derive the probability vector for uncertainty estimation.
• Figure 4: The TRC values on clean data. The higher TRC value the better.
• Figure 5: The performance of FAST with different feature selection strategies.