How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?

TL;DR

The paper investigates how injecting random label noise into gradient updates (label-noise gradient descent) can improve generalization in low signal-to-noise ratio (SNR) regimes. Using an idealized setup with a two-layer convolutional network and a signal-noise data model, it proves that standard gradient descent tends to memorize noise and overfit in low SNR, while label-noise gradient descent suppresses noise memorization and accelerates learning of the true signal, enabling strong generalization even when training loss does not vanish. The authors provide a precise signal-noise decomposition and stage-wise proof sketches, employing a supermartingale argument to bound noise memorization. They corroborate the theory with synthetic experiments and targeted CIFAR-10 experiments showing that the gap in generalization between LN-GD and standard GD widens as the SNR decreases. The work also situates LN-GD relative to sharpness-aware methods, highlighting its simplicity and lack of computational overhead as a practical advantage.

Abstract

The capacity of deep learning models is often large enough to both learn the underlying statistical signal and overfit to noise in the training set. This noise memorization can be harmful especially for data with a low signal-to-noise ratio (SNR), leading to poor generalization. Inspired by prior observations that label noise provides implicit regularization that improves generalization, in this work, we investigate whether introducing label noise to the gradient updates can enhance the test performance of neural network (NN) in the low SNR regime. Specifically, we consider training a two-layer NN with a simple label noise gradient descent (GD) algorithm, in an idealized signal-noise data setting. We prove that adding label noise during training suppresses noise memorization, preventing it from dominating the learning process; consequently, label noise GD enjoys rapid signal growth while the overfitting remains controlled, thereby achieving good generalization despite the low SNR. In contrast, we also show that NN trained with standard GD tends to overfit to noise in the same low SNR setting and establish a non-vanishing lower bound on its test error, thus demonstrating the benefit of introducing label noise in gradient-based training.

Figures (15)

• Figure 1: Test accuracy of VGG-16 on CIFAR-10 with varying SNR. Label noise GD consistently outperforms standard GD, and the gap increases with the noise strength.
• Figure 2: Ratio of noise memorization over signal learning, training loss, and test accuracy, of standard GD and label noise GD. See Section \ref{['sec:sn-decomp']} for definitions of signal learning ($\gamma$) and noise memorization ($\rho$).
• Figure 3: Test accuracy heatmap of Standard GD (left) and Label Noise GD (right) after training.
• Figure 4: Performance of a 3-layer ReLU neural network: The ratio of noise memorization to signal learning, along with training loss and test accuracy, for standard GD and label noise GD.
• Figure 5: Performance on the modified MNIST dataset: The ratio of noise memorization to signal learning, along with training loss and test accuracy, for standard GD and label noise GD.

Theorems & Definitions (45)

• Theorem 1.1: Informal
• Definition 2.1
• Remark 2.2
• Remark 2.3
• Theorem 3.2: GD fails to generalize under low SNR
• Theorem 3.3: Label Noise GD generalizes under low SNR
• Remark 3.4
• Proposition 4.1
• Lemma 4.2
• Lemma 4.3