Gradient Descent as a Shrinkage Operator for Spectral Bias

TL;DR

This work investigates spectral bias in 1D shallow networks by reframing gradient descent as a shrinkage operator on the Jacobian’s singular values, controlled by learning rate $\alpha$ and iterations $q$. The authors derive an explicit masking function $m_{\text{gd}}(s;\alpha,q) = [1 - (1 - \alpha s^{2})^{q}]$ that governs how many frequency components are retained, and connect this to a bandwidth notion via $\rho \approx \sqrt{q}$ and a threshold $\kappa \approx \sqrt{-\log(\epsilon)/(\alpha q)}$. They show activation choices influence the spectral basis: monotonic activations approximate a discrete sine/DST-like basis and GD selectively preserves a set of principal components, while non-monotonic activations rely more on scaling (e.g., sinc) for implicit regularization, with sinc acting as an iteration-efficient surrogate for spectral bias. These insights illuminate how GD hyperparameters and activation types shape generalization in shallow models and suggest avenues for designing activation schemes and training regimens that harness spectral properties in higher dimensions and deeper architectures.

Abstract

We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network's Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias.

Figures (9)

• Figure 1: We argue that the implicit regularization offered by gradient descent (GD) can be viewed as a shrinkage operator on the singular values $\mathbf{s} = [s_{0},\ldots,s_{N}]$ of a network's Jacobian $\nabla f(\mathbf{x})$. The operator applies a masking function $m(s;\alpha,q)$ on the singular values, where $\alpha$ is learning rate and $q$ is the number of iterations which dictate the number $K$ of active principal components. The distribution of $\mathbf{s}$ is activation specific, and therefore how effectively it can be regularized through GD.
• Figure 2: Visual depiction of the masking function $m(s; \alpha, q)$ used for shrinking singular values within GD regularization. We set $\alpha = 1$ so as to ensure the upper singular value of the mask is normalized at unity. We then vary the number of iterations $q$ to demonstrate how this effects the width $\rho$ of the active window.
• Figure 3: Spectrum of the normalized singular values $\mathbf{A} = \eta(\sigma \cdot [\mathbf{x} - \mathbf{b}^{T}])$ for different activations. In (a) one can see that monotonic activations Heaviside -- $(r=1)$, ReLU -- $(r=2)$, and ReLU2 -- $(r=3)$ -- derived in Proposition \ref{['Prop:Act']} -- have different drop off rates related to $r$. tanh has a similar spectrum to Heaviside, whereas GELU and SiLU behave more like ReLU. In contrast to (b) all the monotonic activations enjoy an initial steep drop off, but then plateu. We hypothesize this property to be essential for GD to implicitly regularize for smooth solutions. Conversely, in (b) sinc & Gaussian plateu early, but then see signficant drop off at $k = \sigma$. This is most noticable for sinc as the normalized singular values remain at unity until $k = \sigma$ followed by a precipitous drop off. Since these non-monotonic activations lack the initial steep drop off they instead rely on the scaling $\sigma$ to promote smooth solutions. We hypothesize that the manipulation of GD hyper-parameters (learning rate, number of iterations) will have little to no regularization effect. It should be noted that $\sigma$ has no effect on the spectral distribution of Heaviside, ReLU and ReLU2 as they are all strictly scale equivariant. We arbitrarily use $\sigma$ values of $15$ & $30$ in our visualization.
• Figure 4: Visualization of principal components $u_{k}(x)$ of $\mathbf{A}$ for activation functions (a-d) where $k = [0,4,5,9]$ for $N = M = 5000$. Both (a) Heaviside and (b) tanh exhibit a strict frequency basis, whereas activations like (c) ReLU and (d) ReLU2 have a smooth but less strict frequency basis.
• Figure 5: Visualization of principal components $u_{k}(x)$ of $\mathbf{A}$ for activation functions (a-d) where $k = [0,4,5,9]$ for $N = 5000$ and $M = 50$. In (a) we can see how unsuitable Heaviside is for practical problems where one must generalize smoothly across samples. In contrast, the tanh activation in (b) is able to preserve a smooth frequency basis while dealing with sparse sample observations. The (c) ReLU and (d) ReLU2 activations also exhibit smooth performance, but do not adhere as strictly to a frequency basis. Unlike tanh, however, ReLU and ReLU2 are not scale-sensitive so can always guarantee a smooth basis irrespective of the choice of $\sigma$.

Theorems & Definitions (3)

• Proposition 3.1
• Proposition 4.1
• Proposition 4.2