Class-wise Activation Unravelling the Engima of Deep Double Descent

TL;DR

This work empirically investigates the double descent phenomenon in deep learning by focusing on class-wise activation patterns. It introduces Class Activation Matrices (CAMs) to quantify how class-specific representations evolve with model width and develops a CAM-based richness estimator inspired by Rademacher complexity to measure activation complexity. Through extensive experiments on FCNNs, CNNs, and ResNets across MNIST and CIFAR-10, the authors show that wider networks yield more distinct class activations, that activation richness correlates with the double descent curve, and that over-parameterization can isolate noisy labels in feature space, contributing to improved generalization under label noise. The study synthesizes these findings to offer fresh insights into benign over-parameterization and outlines directions for theoretical development and broader applicability.

Abstract

Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed for this phenomenon in specific contexts, an accepted theory for its occurring mechanism in deep learning remains yet to be established. In this study, we revisited the phenomenon of double descent and discussed the conditions of its occurrence. This paper introduces the concept of class-activation matrices and a methodology for estimating the effective complexity of functions, on which we unveil that over-parameterized models exhibit more distinct and simpler class patterns in hidden activations compared to under-parameterized ones. We further looked into the interpolation of noisy labelled data among clean representations and demonstrated overfitting w.r.t. expressive capacity. By comprehensively analysing hypotheses and presenting corresponding empirical evidence that either validates or contradicts these hypotheses, we aim to provide fresh insights into the phenomenon of double descent and benign over-parameterization and facilitate future explorations. By comprehensively studying different hypotheses and the corresponding empirical evidence either supports or challenges these hypotheses, our goal is to offer new insights into the phenomena of double descent and benign over-parameterization, thereby enabling further explorations in the field. The source code is available at https://github.com/Yufei-Gu-451/sparse-generalization.git.

Figures (26)

• Figure 1: The generalization performance of two-layer FCNNs trained on MNIST $(N=4000)$, under varying explicit label noise ratios of $p = [0\%, 10\%, 20\%]$. Optimized with SGD for 4000 epochs and decreasing learning rate. The test error curve with no external label noise is monotonic over the increase of model width $k$. The test error curve with external label noise performs the double descent phenomenon which decreases and increases and decreases again after the interpolation threshold.
• Figure 2: The generalization performance of five-layer CNNs trained on CIFAR-10 ($N=50000$), under varying explicit label noise ratios of $p = [0\%, 10\%, 20\%]$. Optimized with SGD for 200 epochs and decreasing learning rate. All test error curve performs the double descent phenomenon which decreases and increases and decreases again after the interpolation threshold.
• Figure 3: The generalization performance of ResNet18s trained on CIFAR-10 ($N=50000$), under varying explicit label noise ratios of $p = [0\%, 10\%, 20\%]$. Optimized with SGD for 200 epochs and decreasing learning rate. All test error curve performs the double descent phenomenon which decreases and increases and decreases again after the interpolation threshold.
• Figure 4: The Heatmap of the cosine similarities between every pair of the computed CAMs, drawn from one two-layer FCNN trained on MNIST with no label noise introduced as an example. Four heatmaps are selected representing stages of model parameterization (Hidden Units $k \in [10, 20, 40, 200]$). The first row of heatmaps demonstrated the similarities between the Input Layer (layer $f_0$) and the Hidden Layer (layer $f_1$). The second row of heatmaps demonstrated the similarities between the Hidden Layer (layer $f_1$) and the Output Layer (layer $f_2$). The half bottom-left section of the heatmap matrix was left as 0 and left blank.
• Figure 5: The phenomenon of double descent on two-layer FCNNs trained on MNIST $(N=4000)$, under varying explicit label noise ratios of $p = [0\%, 10\%, 20\%]$ and the mean similarities among all Class Activation Matrices (CAMs). The experimental results are placed at the top for easier comparison. The yellow line denotes the activation similarities between the input layer (layer $f_0$) and the hidden layer (layer $f_1$), whereas the purple line signifies the similarities between the hidden layer (layer $f_1$) and the output layer (layer $f_2$).