Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks

Jiahao Lu

TL;DR

The paper reframes regularization through a spectral lens, introducing the Spectral Suppression Ratio (SSR) and a discrete radial profiling method to quantify how $L2$ and Dropout bias CNNs toward low-frequency features. By treating CNN weights as 2D signals and tracking their frequency evolution during training, the study demonstrates that $L2$ acts as a physical low-pass filter, suppressing high-frequency energy by roughly a factor of 3 and correlating with higher test accuracy on CIFAR-10 with ResNet-18. A critical accuracy–robustness trade-off emerges: while low-frequency specialization improves robustness to high-frequency information loss (e.g., blur, low resolution), it increases vulnerability to broadband noise, revealing a nuanced, application-dependent regularization strategy. The work provides a practical, signal-processing perspective on generalization, linking spectral dynamics to generalization performance and proposing directions for frequency-aware regularization and theoretical grounding in NTK-related analyses.

Abstract

Regularization techniques such as L2 regularization (Weight Decay) and Dropout are fundamental to training deep neural networks, yet their underlying physical mechanisms regarding feature frequency selection remain poorly understood. In this work, we investigate the Spectral Bias of modern Convolutional Neural Networks (CNNs). We introduce a Visual Diagnostic Framework to track the dynamic evolution of weight frequencies during training and propose a novel metric, the Spectral Suppression Ratio (SSR), to quantify the "low-pass filtering" intensity of different regularizers. By addressing the aliasing issue in small kernels (e.g., 3x3) through discrete radial profiling, our empirical results on ResNet-18 and CIFAR-10 demonstrate that L2 regularization suppresses high-frequency energy accumulation by over 3x compared to unregularized baselines. Furthermore, we reveal a critical Accuracy-Robustness Trade-off: while L2 models are sensitive to broadband Gaussian noise due to over-specialization in low frequencies, they exhibit superior robustness against high-frequency information loss (e.g., low resolution), outperforming baselines by >6% in blurred scenarios. This work provides a signal-processing perspective on generalization, confirming that regularization enforces a strong spectral inductive bias towards low-frequency structures.

Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks

TL;DR

The paper reframes regularization through a spectral lens, introducing the Spectral Suppression Ratio (SSR) and a discrete radial profiling method to quantify how and Dropout bias CNNs toward low-frequency features. By treating CNN weights as 2D signals and tracking their frequency evolution during training, the study demonstrates that acts as a physical low-pass filter, suppressing high-frequency energy by roughly a factor of 3 and correlating with higher test accuracy on CIFAR-10 with ResNet-18. A critical accuracy–robustness trade-off emerges: while low-frequency specialization improves robustness to high-frequency information loss (e.g., blur, low resolution), it increases vulnerability to broadband noise, revealing a nuanced, application-dependent regularization strategy. The work provides a practical, signal-processing perspective on generalization, linking spectral dynamics to generalization performance and proposing directions for frequency-aware regularization and theoretical grounding in NTK-related analyses.

Abstract

Regularization techniques such as L2 regularization (Weight Decay) and Dropout are fundamental to training deep neural networks, yet their underlying physical mechanisms regarding feature frequency selection remain poorly understood. In this work, we investigate the Spectral Bias of modern Convolutional Neural Networks (CNNs). We introduce a Visual Diagnostic Framework to track the dynamic evolution of weight frequencies during training and propose a novel metric, the Spectral Suppression Ratio (SSR), to quantify the "low-pass filtering" intensity of different regularizers. By addressing the aliasing issue in small kernels (e.g., 3x3) through discrete radial profiling, our empirical results on ResNet-18 and CIFAR-10 demonstrate that L2 regularization suppresses high-frequency energy accumulation by over 3x compared to unregularized baselines. Furthermore, we reveal a critical Accuracy-Robustness Trade-off: while L2 models are sensitive to broadband Gaussian noise due to over-specialization in low frequencies, they exhibit superior robustness against high-frequency information loss (e.g., low resolution), outperforming baselines by >6% in blurred scenarios. This work provides a signal-processing perspective on generalization, confirming that regularization enforces a strong spectral inductive bias towards low-frequency structures.
Paper Structure (26 sections, 3 equations, 5 figures, 2 tables)

This paper contains 26 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Regularization and Generalization.
  4. Spectral Bias in Neural Networks.
  5. Frequency and Robustness.
  6. The Generalization Puzzle.
  7. Method
  8. Preliminaries: Spectral Representation of Weights
  9. Algorithm: Discrete Radial Profiling for Small Kernels
  10. Metric: The Spectral Suppression Ratio (SSR)
  11. Implementation and Experimental Setup
  12. Preliminary Study: Synthetic Signal Fitting
  13. Experimental Setup
  14. Results: Frequency-Selective Suppression
  15. Experiments
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Frequency Learning Curves. Left (Baseline): High-frequency noise (red) accumulates steadily as training proceeds. Center (Moderate L2): High-frequency components are suppressed (flat red line) while low-frequency signal (blue) remains strong—demonstrating selective filtering. Right (Strong L2): Over-regularization suppresses all frequencies, collapsing low-frequency signal and causing underfitting.
  • Figure 2: Spectral Evolution of ResNet-18 Weights. The x-axis represents training epochs, and the y-axis represents frequency (Bottom=Low, Top=High). L2 regularization (Center) effectively suppresses high-frequency energy accumulation compared to Baseline (Left).
  • Figure 3: Spectral Suppression Ratio (SSR). Higher bars (closer to 0) indicate stronger suppression of high-frequency components. L2 demonstrates the strongest filtering effect.
  • Figure 4: Low Resolution (Advantage). L2 (Green) outperforms Baseline.
  • Figure 5: Gaussian Noise (Disadvantage). L2 (Green) degrades faster.