Generalization ability and Vulnerabilities to adversarial perturbations: Two sides of the same coin

TL;DR

This work uses the self-organizing map (SOM) to analyze DL models' internal codes associated with DNNs' decision-making and suggests that shallow layers close to the input layer map onto homogeneous codes and that deep layers close to the output layer transform these homogeneous codes in shallow layers to diverse codes.

Abstract

Deep neural networks (DNNs), the agents of deep learning (DL), require a massive number of parallel/sequential operations, which makes it difficult to comprehend them and impedes proper diagnosis. Without better knowledge of DNNs' internal process, deploying DNNs in high-stakes domains may lead to catastrophic failures. Therefore, to build more reliable DNNs/DL, it is imperative that we gain insights into their underlying decision-making process. Here, we use the self-organizing map (SOM) to analyze DL models' internal codes associated with DNNs' decision-making. Our analyses suggest that shallow layers close to the input layer map onto homogeneous codes and that deep layers close to the output layer transform these homogeneous codes in shallow layers to diverse codes. We also found evidence indicating that homogeneous codes may underlie DNNs' vulnerabilities to adversarial perturbations.

Figures (16)

• Figure 1: Schematics of SOM-based analysis. We display ResNet18 as an example. The same approach is applied to ResNet50, VGG19 and DenseNet121. We convert individual feature maps’ responses to vector codes using max, mean, standard deviation and center of mass. Vector codes are abstract representations of feature map outputs (hidden layer representations) and are inputs to a SOM, which is used to identify the functional codes.
• Figure 1: Density of BMUs. Four rows show BMUs of ResNet18, ResNet50, VGG19 and DenseNet121, respectively. Each column shows BMUs from one of 4 CLs (see the name of CL shown above the column). We note that the density per area is very low. For illustration purpose only, we scaled the density by 100 times.
• Figure 2: Unfolding SOM. When Euclidean distances between two SOM units or the class centers are estimated, all BMUs are transferred to the unfolded SOM. (A), Torus distance. To estimate the torus distance, all BMUs are mapped onto 9 identical grids. (B), Examples of BMUs in the original (top row) and unfolded (bottom row) SOMs. The columns show BMUs from CL1, CL2, CL3 and CL4, respectively. The class center is the center of BMUs’ mass in the unfolded SOMs shown in the bottom row. It should be noted that SOMs in the bottom row are three times larger than the original SOMs in the top row.
• Figure 2: BMUs of vector codes obtained from ResNet50. The panels show BMUs of SOMs trained on ResNet50. (A), BMUs of vector codes in CL1. The top row shows the BMUs of SOM-tr, whereas the bottom row shows the BMUs of SOM-val. Each column shows BMUs of the inputs in the same class. (B), the same as (A), but vector codes were obtained from CL2. (C), the same as (A), but vector codes were obtained from CL3.(D), the same as (A), but vector codes were obtained from CL4.
• Figure 3: Loss function of SOMs trained on 4 models' vector codes. Blue, orange, green and red line denote CL1, CL2, CL3 and CL4, respectively. (A)-(D), Time courses of SOM training of ResNet18, ResNet50, DenseNet121 and VGG19.