Context-Aware Fuzzing for Robustness Enhancement of Deep Learning Models
Haipeng Wang, Zhengyuan Wei, Qilin Zhou, Wing-Kwong Chan
TL;DR
The paper introduces Contextual Confidence (CC), a black-box, non-coverage-based metric for evaluating the representativeness of test cases, and Clover, a context-aware fuzzing framework designed to improve the robustness of deep learning models within a testing-retraining pipeline. Clover leverages seed equivalence and adversarial front objects (α- and β-AFOs) to generate test cases and constructs a robustness-focused test suite through a layered, CC-driven selection process. Empirical results across multiple datasets and models show that Clover outperforms state-of-the-art selection and fuzzing techniques (Adapt and RobOT) in both the quantity and quality of test cases and in robustness improvements after retraining, with configuration using CC generally outperforming configurations based on DeepGini or FOL. The work demonstrates the practical value of context-aware fuzzing for robustness and outlines future directions, including refinements to seed equivalence and broader validation across architectures and defenses.
Abstract
In the testing-retraining pipeline for enhancing the robustness property of deep learning (DL) models, many state-of-the-art robustness-oriented fuzzing techniques are metric-oriented. The pipeline generates adversarial examples as test cases via such a DL testing technique and retrains the DL model under test with test suites that contain these test cases. On the one hand, the strategies of these fuzzing techniques tightly integrate the key characteristics of their testing metrics. On the other hand, they are often unaware of whether their generated test cases are different from the samples surrounding these test cases and whether there are relevant test cases of other seeds when generating the current one. We propose a novel testing metric called Contextual Confidence (CC). CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Based on this metric, we further propose a novel fuzzing technique Clover as a DL testing technique for the pipeline. In each fuzzing round, Clover first finds a set of seeds whose labels are the same as the label of the seed under fuzzing. At the same time, it locates the corresponding test case that achieves the highest CC values among the existing test cases of each seed in this set of seeds and shares the same prediction label as the existing test case of the seed under fuzzing that achieves the highest CC value. Clover computes the piece of difference between each such pair of a seed and a test case. It incrementally applies these pieces of differences to perturb the current test case of the seed under fuzzing that achieves the highest CC value and to perturb the resulting samples along the gradient to generate new test cases for the seed under fuzzing.