Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks

Julia Nakhleh, Robert D. Nowak

TL;DR

This work addresses the problem of finding the sparsest ReLU interpolant for given data by introducing a differentiable objective based on the $\ell^p$ quasinorm with $0<p<1$, whose global minima correspond to sparsest single-hidden-layer networks. A variational reformulation recasts the problem as optimizing continuous piecewise-linear functions with respect to a $p$-variation cost $V_p(f)$ (and $V_0(f)$ counting knots), enabling a gradient-based approach to sparse interpolation. The authors establish univariate results showing uniqueness and explicit sparsity bounds, and extend to multivariate inputs by proving that sufficiently small $p$ yields sparsest solutions with width-invariant neuron counts and $O(N)$ active parameters; a finite-dimensional activation-pattern reformulation underpins these results. Experiments with reweighted $\ell^1$ regularization corroborate the theoretical claims, demonstrating faster and sparser interpolation than standard regularization schemes. Overall, the paper provides a principled continuous route to recovering truly sparse ReLU networks without pruning, with broad implications for efficiency and interpretability.

Abstract

Overparameterized neural networks can interpolate a given dataset in many different ways, prompting the fundamental question: which among these solutions should we prefer, and what explicit regularization strategies will provably yield these solutions? This paper addresses the challenge of finding the sparsest interpolating ReLU network--i.e., the network with the fewest nonzero parameters or neurons--a goal with wide-ranging implications for efficiency, generalization, interpretability, theory, and model compression. Unlike post hoc pruning approaches, we propose a continuous, almost-everywhere differentiable training objective whose global minima are guaranteed to correspond to the sparsest single-hidden-layer ReLU networks that fit the data. This result marks a conceptual advance: it recasts the combinatorial problem of sparse interpolation as a smooth optimization task, potentially enabling the use of gradient-based training methods. Our objective is based on minimizing $\ell^p$ quasinorms of the weights for $0 < p < 1$, a classical sparsity-promoting strategy in finite-dimensional settings. However, applying these ideas to neural networks presents new challenges: the function class is infinite-dimensional, and the weights are learned using a highly nonconvex objective. We prove that, under our formulation, global minimizers correspond exactly to sparsest solutions. Our work lays a foundation for understanding when and how continuous sparsity-inducing objectives can be leveraged to recover sparse networks through training.

Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks

TL;DR

This work addresses the problem of finding the sparsest ReLU interpolant for given data by introducing a differentiable objective based on the quasinorm with , whose global minima correspond to sparsest single-hidden-layer networks. A variational reformulation recasts the problem as optimizing continuous piecewise-linear functions with respect to a -variation cost (and counting knots), enabling a gradient-based approach to sparse interpolation. The authors establish univariate results showing uniqueness and explicit sparsity bounds, and extend to multivariate inputs by proving that sufficiently small yields sparsest solutions with width-invariant neuron counts and active parameters; a finite-dimensional activation-pattern reformulation underpins these results. Experiments with reweighted regularization corroborate the theoretical claims, demonstrating faster and sparser interpolation than standard regularization schemes. Overall, the paper provides a principled continuous route to recovering truly sparse ReLU networks without pruning, with broad implications for efficiency and interpretability.

Abstract

Overparameterized neural networks can interpolate a given dataset in many different ways, prompting the fundamental question: which among these solutions should we prefer, and what explicit regularization strategies will provably yield these solutions? This paper addresses the challenge of finding the sparsest interpolating ReLU network--i.e., the network with the fewest nonzero parameters or neurons--a goal with wide-ranging implications for efficiency, generalization, interpretability, theory, and model compression. Unlike post hoc pruning approaches, we propose a continuous, almost-everywhere differentiable training objective whose global minima are guaranteed to correspond to the sparsest single-hidden-layer ReLU networks that fit the data. This result marks a conceptual advance: it recasts the combinatorial problem of sparse interpolation as a smooth optimization task, potentially enabling the use of gradient-based training methods. Our objective is based on minimizing quasinorms of the weights for , a classical sparsity-promoting strategy in finite-dimensional settings. However, applying these ideas to neural networks presents new challenges: the function class is infinite-dimensional, and the weights are learned using a highly nonconvex objective. We prove that, under our formulation, global minimizers correspond exactly to sparsest solutions. Our work lays a foundation for understanding when and how continuous sparsity-inducing objectives can be leveraged to recover sparse networks through training.

Paper Structure

This paper contains 32 sections, 10 theorems, 63 equations, 20 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Sparsity via $\ell^p$ minimization in finite-dimensional linear models:
  4. $\ell^1$ path norm regularization in single-layer ReLU networks:
  5. Empirical methods for training sparse neural networks:
  6. Provable sparsest-recovery in specialized neural network settings:
  7. Univariate $\ell^p$-regularized neural networks
  8. Variational reformulation of \ref{['opt:min_p_NN']} and \ref{['opt:min_0_NN']}
  9. Geometric characterization of solutions to \ref{['opt:min_p_f']}
  10. Uniqueness and sparsity of solutions to \ref{['opt:min_p_f']} for $0 < p < 1$
  11. Multivariate $\ell^p$-regularized neural networks
  12. Experiments
  13. Conclusion and Discussion
  14. Proofs of main results
  15. Univariate results
  16. ...and 17 more sections

Key Result

Proposition 3.1

For any $0 \leq p \leq 1$, the set $S_p^*$ is exactly the set of minimizers of where the optimization in opt:min_p_f is taken over all $f: \mathbb{R} \to \mathbb{R}$ which are continuous piecewise linear (CPWL) with at most $K$ knots. For such CPWL functions $f$, we define with the first $\sup$ taken over all partitions ${\mathcal{P}} = \{x_0 < \dots < x_{n_{{\mathcal{P}}}} \}$ of $\mathbb{R}$,

Figures (20)

  • Figure 1: \ref{['fig:l1_sparse_nonsparse']} shows several univariate min-$\ell^1$ path norm interpolants of a given dataset. Such solutions are generally non-unique, and always include at least one sparsest interpolant (black), but also include arbitrarily non-sparse interpolants (blue, red, green). \ref{['fig:2d_ridge', 'fig:2d_cc']}: two different ReLU network interpolants of a the same 2D dataset with different numbers of active neurons and parameters. \ref{['fig:2d_ridge']} has 5 nonzero input weight/bias parameters (its $\ell^0$ path norm as in \ref{['opt:min_0_NN_multi']}), while \ref{['fig:2d_cc']} has 16.
  • Figure 2: Illustration of \ref{['th:geom_char']}. By \ref{['th:geom_char']},\ref{['th:geom_char_1']}, any $f \in S_p^*$ for $0 < p < 1$ must agree with the function in \ref{['fig:geom_char_sol0']} on $(-\infty, x_7]$ and $[x_{10}, \infty)$. The only possible ambiguity occurs between $x_7$ and $x_{10}$, where all points have the same discrete curvature. Here the function behavior is described by \ref{['th:geom_char']},\ref{['th:geom_char_2b']}. \ref{['fig:geom_char_sol1', 'fig:geom_char_sol2']} show two other functions whose behavior on $[x_7, x_{10}]$ also concurs with \ref{['th:geom_char']},\ref{['th:geom_char_2b']}.
  • Figure 3: Sparsity over time of networks trained to interpolation with a reweighted $\ell^1$ algorithm (see \ref{['appendix:rw_l1']}) for $\ell^p$ path norm regularization, $p \in \{0.4, 0.7, 1 \}$, and of unregularized and weight decay-regularized networks. Results on the left are for a synthetic univariate "peak/plateau" dataset, and results on the right are for a high-dimensional set of random data and labels. The gray dashed lines reflect the true minimal sparsity (in the univariate case, left) and the upper bound on the minimal sparsity guaranteed by \ref{['prop:width_invariance_multivar']} in the multivariate case (right). For further details, results, and discussion, see \ref{['appendix:experimental_setup_results']}.
  • Figure 4: Left: Illustration of the case $\mathop{\mathrm{sgn}}\nolimits \left( s_i - s_{\mathrm{in}}(f, x_i) \right) = \mathop{\mathrm{sgn}}\nolimits \left(s_{\mathrm{out}}(f, x_{i+1}) -s_i\right)$ addressed in \ref{['lemma:same_sign']}. Right: illustration of the case $\mathop{\mathrm{sgn}}\nolimits \left( s_i - s_{\mathrm{in}}(f, x_i) \right) \neq \mathop{\mathrm{sgn}}\nolimits \left(s_{\mathrm{out}}(f, x_{i+1}) -s_i\right)$ addressed in \ref{['lemma:opp_sign']}. In both cases, the functions in black have strictly greater $V_p$ for $0 \leq p < 1$ than the functions in blue.
  • Figure 5: Base case of \ref{['lemma:opp_sign']}, where we consider the possibility that $f \in S_p^*$ for some $0 \leq p < 1$ has a single knot at some $x \in (x_i, x_{i+1})$ where $\mathop{\mathrm{sgn}}\nolimits(a-b) \neq \mathop{\mathrm{sgn}}\nolimits(b-c)$. Here $\tau := \frac{x - x_i}{x_{i+1}-x_i}$.
  • ...and 15 more figures

Theorems & Definitions (20)

  • Proposition 3.1
  • Remark 1
  • Theorem 3.1
  • Corollary 3.1.1
  • Corollary 3.1.2
  • Theorem 3.2
  • Proposition 4.1
  • Lemma 4.1
  • Theorem 4.1
  • Remark 2
  • ...and 10 more