Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples

TL;DR

The paper addresses interpretability in deep neural networks by analyzing internal representations under adversarial perturbations, revealing that high-level neurons do not reliably detect semantic objects and that deep representations are not robust concepts. It introduces an ensemble-optimization adversarial dataset and metrics (including a WordNet-based LC and distance-based CS/r metrics) to quantify the inconsistency between real and adversarial representations. To improve interpretability, the authors propose adversarial training with a consistent loss that aligns representations of real and adversarial inputs, enabling tracing predictions to influential neurons via a prediction-difference metric and improving robustness against adversarial attacks. The work demonstrates that interpretability can be enhanced with a modest drop in real-data accuracy and offers practical mechanisms to detect adversarial inputs and explain model decisions, with implications for safer, more transparent AI systems.

Abstract

Deep neural networks (DNNs) have demonstrated impressive performance on a wide array of tasks, but they are usually considered opaque since internal structure and learned parameters are not interpretable. In this paper, we re-examine the internal representations of DNNs using adversarial images, which are generated by an ensemble-optimization algorithm. We find that: (1) the neurons in DNNs do not truly detect semantic objects/parts, but respond to objects/parts only as recurrent discriminative patches; (2) deep visual representations are not robust distributed codes of visual concepts because the representations of adversarial images are largely not consistent with those of real images, although they have similar visual appearance, both of which are different from previous findings. To further improve the interpretability of DNNs, we propose an adversarial training scheme with a consistent loss such that the neurons are endowed with human-interpretable concepts. The induced interpretable representations enable us to trace eventual outcomes back to influential neurons. Therefore, human users can know how the models make predictions, as well as when and why they make errors.

Figures (15)

• Figure 1: The overall framework in this paper. (a) We first generate a set of $500$K adversarial images by the ensemble-optimization attack and re-examine the visual representations with the conclusions of dummy object/part detectors and inconsistent visual representations. (b) We show that adversarial training facilitates the improvement of the interpretability and consistency of representations in DNNs. (c) The interpretable representations enable us to trace the eventual predictions back to influential neurons, and explain how the predictions have been made as well as when and why an error occurs.
• Figure 2: The real and adversarial images with highest activations for neurons in VGG-16 pool5 layer. The neurons have explicit semantic meanings in real images, which do not appear in adversarial images. The adversarial images in red boxes have the target classes the same as the meanings of the neurons (e.g., the model misclassifies the adversarial images in (a) as birds). The highlighted regions are found by discrepancy map zhou2014object. More visualization results of AlexNet and ResNet-18 can be found in Appendix.
• Figure 3: Illustration for quantifying the level and the consistency of features on WordNet. The red path indicates the distance between persian cat and wolfhound$d=4$, which is larger than the distance between tabby cat and kitty cat ($d=2$ indicated by the green path).
• Figure 4: Cosine similarity between the classes of real and adversarial images for neurons against the level and the consistency of their features. We average the $\mathrm{CS}_1$ and $\mathrm{CS}_2$ of different neurons around a given $\mathrm{LC}$ value. The neurons come from all convolutional layers in each model.
• Figure 5: The distribution of adversarial images against the ratio of distance. The vertical axis shows the number of adversarial images around a given ratio. The average $r_1$ of AlexNet, VGG-16 and ResNet-18 are $1.07$, $1.26$ and $1.34$ for all adversarial images while the average $r_2$ of them are $0.80$, $0.73$ and $0.76$, respectively.