Deep Feature Response Discriminative Calibration

TL;DR

This work proposes a plugin-based calibration module incorporated into a modified ResNet architecture, termed Response Calibration Networks (ResCNet), and proposes a method that discriminatively calibrates feature responses to improve the feature discriminability of the neural feature response.

Abstract

Deep neural networks (DNNs) have numerous applications across various domains. Several optimization techniques, such as ResNet and SENet, have been proposed to improve model accuracy. These techniques improve the model performance by adjusting or calibrating feature responses according to a uniform standard. However, they lack the discriminative calibration for different features, thereby introducing limitations in the model output. Therefore, we propose a method that discriminatively calibrates feature responses. The preliminary experimental results indicate that the neural feature response follows a Gaussian distribution. Consequently, we compute confidence values by employing the Gaussian probability density function, and then integrate these values with the original response values. The objective of this integration is to improve the feature discriminability of the neural feature response. Based on the calibration values, we propose a plugin-based calibration module incorporated into a modified ResNet architecture, termed Response Calibration Networks (ResCNet). Extensive experiments on datasets like CIFAR-10, CIFAR-100, SVHN, and ImageNet demonstrate the effectiveness of the proposed approach. The developed code is publicly available at https://github.com/tcmyxc/ResCNet.

Figures (6)

• Figure 1: Module architecture. From the module architecture, it can be seen that SENet further scales the features extracted by the ResNet residual branches, whereas our proposed architecture provides additional calibration of these features.
• Figure 2: The distribution of response values after model convergence using ResNet-32 on CIFAR-100 dataset. From the outer contour of the violin plot, it can be observed that the response values of a single convolutional kernel (neuron) roughly follow a Gaussian distribution.
• Figure 3: ResNet block and ResCNet block.
• Figure 4: Top-1 error of ImageNet using ResNet and ResCNet. From the training curves, it can be observed that the error rate of the ResCNet model is lower than that of the ResNet model.
• Figure 5: Comparison of response value distributions. We compared the response value distributions of the ResNet-32, SENet-32, and ResCNet-32 models on the CIFAR-100 dataset. The red dashed bounding boxes serve as visual indicators of the enhanced discriminability in neural feature responses.