More is Less: Inducing Sparsity via Overparameterization

TL;DR

This paper analyzes why overparameterized neural models can generalize well by studying sparse recovery as a tractable proxy. It proves that gradient flow on a deeply factorized, overparameterized square loss converges to a near-minimal $\ell_1$-norm solution among all feasible solutions, with depth $L$ inducing a transition from standard to weighted $\ell_1$ regularization. The results yield near-optimal sample complexity for compressed sensing, showing recovery with $M \sim s \log(N/s)$ measurements under common matrix properties and establishing robustness to noise via NSP and quotient properties. The findings illuminate an implicit bias mechanism that favors sparse representations and offer practical implications for CS and beyond, supported by numerical experiments that align with theory and demonstrate depth- and initialization-dependent behavior.

Abstract

In deep learning it is common to overparameterize neural networks, that is, to use more parameters than training samples. Quite surprisingly training the neural network via (stochastic) gradient descent leads to models that generalize very well, while classical statistics would suggest overfitting. In order to gain understanding of this implicit bias phenomenon we study the special case of sparse recovery (compressed sensing) which is of interest on its own. More precisely, in order to reconstruct a vector from underdetermined linear measurements, we introduce a corresponding overparameterized square loss functional, where the vector to be reconstructed is deeply factorized into several vectors. We show that, if there exists an exact solution, vanilla gradient flow for the overparameterized loss functional converges to a good approximation of the solution of minimal $\ell_1$-norm. The latter is well-known to promote sparse solutions. As a by-product, our results significantly improve the sample complexity for compressed sensing via gradient flow/descent on overparameterized models derived in previous works. The theory accurately predicts the recovery rate in numerical experiments. Our proof relies on analyzing a certain Bregman divergence of the flow. This bypasses the obstacles caused by non-convexity and should be of independent interest.

Figures (4)

• Figure 1: We compare the recovery probability for different method via heatmaps. The horizontal axis is the sparsity level $s$, and vertical axis is the number of measurements $M$. The result (a) shows that we cannot expect any recovery by using the naive quadratic loss. Note that our model (c) achieves similar, and in fact arguably better performance than the well known (b) basis pursuit method.
• Figure 2: ${\bf x}_*\in\mathbb{R}_+^{N}$: Comparison of recovery probability between different optimization method.
• Figure 3: ${\bf x}_*\in\mathbb{R}^{N}$: Comparison of recovery probability between different optimization method.
• Figure 4: Comparison of initialization scaling. Setting: $N=1000$, $M=150$. Axes: y-axis = $\varepsilon$ defined in \ref{['eq:exp_setting2']}, x-axis = $\alpha$ where $\tilde{{\bf u}}_0=\tilde{{\bf v}}_0=\alpha{\bf 1}$. We can see that for $L>2$ the error decreases polynomial, while for $L=2$ the scaling is only logarithmic.

Theorems & Definitions (25)

• Theorem 1.1
• Remark 1.2
• Corollary 1.3
• Theorem 1.4
• Theorem 2.1: Equivalence to $\ell_1$-minimization, positive case
• Theorem 2.2
• Definition 2.3: Bregman Divergence
• Lemma 2.4: Bregman1967
• Lemma 2.5
• proof