Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfitting

TL;DR

The paper investigates epoch-wise double descent under label noise using CIFAR-10 and simple fully connected nets. It decomposes training dynamics into clean- and noisy-data contributions to reveal distinct learning phases, internal signal separation, and the emergence of a large activation in shallow layers that correlates with input patterns. The authors propose a novel mechanism linking deep double descent, benign overfitting, and large activation, suggesting a compression-based subnetwork formation that enables robust generalization despite noisy labels. These findings offer a unified viewpoint on how deep networks can generalize well in noisy environments and provide a simple scenario for understanding deep double descent.

Abstract

Deep double descent is one of the key phenomena underlying the generalization capability of deep learning models. In this study, epoch-wise double descent, which is delayed generalization following overfitting, was empirically investigated by focusing on the evolution of internal structures. Fully connected neural networks of three different sizes were trained on the CIFAR-10 dataset with 30% label noise. By decomposing the loss curves into signal contributions from clean and noisy training data, the epoch-wise evolutions of internal signals were analyzed separately. Three main findings were obtained from this analysis. First, the model achieved strong re-generalization on test data even after perfectly fitting noisy training data during the double descent phase, corresponding to a "benign overfitting" state. Second, noisy data were learned after clean data, and as learning progressed, their corresponding internal activations became increasingly separated in outer layers; this enabled the model to overfit only noisy data. Third, a single, very large activation emerged in the shallow layer across all models; this phenomenon is referred as "outliers," "massive activa-tions," and "super activations" in recent large language models and evolves with re-generalization. The magnitude of large activation correlated with input patterns but not with output patterns. These empirical findings directly link the recent key phenomena of "deep double descent," "benign overfitting," and "large activation", and support the proposal of a novel scenario for understanding deep double descent.

Figures (9)

• Figure 1: Accuracy and loss curves during training. (A)--(C) Accuracy curves and (D)--(F) loss curves. Each column represents a different model: (A) and (D) for MLP7, (B) and (E) for MLP5, and (C) and (F) for MLP3. Training curves are decomposed into two components: "clean training" and "noisy training". The accuracy and loss values are evaluated on clean training data during "clean training" and on noisy training data during "noisy training." "Noisy training" is further evaluated using two different label sets: assigned noisy labels ("noisy training / noisy") and original clean labels ("noisy training / clean"). Vertical dashed lines and shaded background delineate distinct training phases from their behaviors. Specifically, training phase for MLP7 is divided into three phases: an initial phase where the model fits only clean training data (1--100 epochs), a middle phase where the model fits noisy training data and clean training data, resulting in overfitting (100--7,000 epochs), and a final phase where double descent occurs (7,000--100,000 epochs). In contrast, those for MLP5 and MLP3 are divided into two phases: an initial phase (1--100 epochs for MLP5 and 1--300 epochs for MLP3) and a final phase (100--1,000,000 epochs for MLP5 and 300--1,000,000 epochs for MLP3).
• Figure 2: Data grouping methods for cosine similarity analysis using the "frog" class as an example. "Clean data" contain frog images labeled as "frog," whereas "noisy data" contain frog images with non "frog" labels.
• Figure 3: Epoch-wise evolution of cosine similarity between the mean activations of hidden layers from clean and noisy training data for MLP7, MLP5, and MLP3 from left to right. Vertical dashed lines and background colors indicate distinct training phases, consistent with those in Fig. \ref{['fig:loss_acc']}. The hidden-layer activations become increasingly separable in outer layers as the training progresses. MLP3 shows a weaker degree of separation, whereas MLP7 and MLP5 show comparable degrees of separations in the outer layers.
• Figure 4: Epoch-wise evolution of cosine similarity between the mean activations of hidden layers for correctly predicted test data and clean training data (top row) and those for correctly predicted test data and noisy training data (bottom row). Each column represents a different model: (A) and (D) are for MLP7, (B) and (E) for MLP5, and (C) and (F) for MLP3. The correctly predicted test data signals are closely aligned with those of clean training data signals for all models, whereas these are separated from noisy training data signals in MLP7 and MLP5.
• Figure 5: Same as Fig. \ref{['fig:correct_test']} but for incorrectly predicted test data.