A Dual Certificate Approach to Sparsity in Infinite-Width Shallow Neural Networks

Abstract

In this paper, we study total variation (TV)-regularized training of infinite-width shallow ReLU neural networks, formulated as a convex optimization problem over measures on the unit sphere. Our approach leverages the duality theory of TV-regularized optimization problems to establish rigorous guarantees on the sparsity of the solutions to the training problem. Our analysis further characterizes how and when this sparsity persists in a low noise regime and for small regularization parameter. The key observation that motivates our analysis is that, for ReLU activations, the associated dual certificate is piecewise linear in the weight space. Its linearity regions, which we name dual regions, are determined by the activation patterns of the data via the induced hyperplane arrangement. Taking advantage of this structure, we prove that, on each dual region, the dual certificate admits at most one extreme value. As a consequence, the support of any minimizer is finite, and its cardinality can be bounded from above by a constant depending only on the geometry of the data-induced hyperplane arrangement. Then, we further investigate sufficient conditions ensuring uniqueness of such sparse solution. Finally, under a suitable non-degeneracy condition on the dual certificate along the boundaries of the dual regions, we prove that in the presence of low label noise and for small regularization parameter, solutions to the training problem remain sparse with the same number of Dirac deltas. Additionally, their location and the amplitudes converge, and, in case the locations lie in the interior of a dual region, the convergence happens with a rate that depends linearly on the noise and the regularization parameter.

Figures (6)

• Figure 1: Dual regions on $\mathbb S^1$ induced by two inputs ($d=n=2$).
• Figure 2: Dual region decomposition of $\mathbb S^1$ and optimal solution $\mu_{\lambda,\zeta}$. The dashed diameters correspond to the boundaries $\langle w,x_j\rangle=0$ induced by the five inputs \ref{['eq:data_points']}, partitioning $\mathbb S^1$ into sectors on which $(\mathbf 1\{\langle w,x_j\rangle>0\})_{j=1}^5$ is constant. We display the recovered atoms $\{w_i\}_{i=1}^3$ of $\mu_{\lambda,\zeta}$. The arrow corresponds to the vector $\frac{\tilde{z}_{\lambda,\zeta}}{\|\tilde{z}_{\lambda,\zeta}\|}$ as in \ref{['eq:gated_vector_2']} with $\tilde{z}_{\lambda,\zeta}$ defined in \ref{['eq:gated_vector_1']}. As predicted by Lemma \ref{['unique_c.p.soft']}, this vector aligns with the location of the Dirac delta of the solution $\mu_{\lambda,\zeta}$ belonging to the interior of a dual region.
• Figure 3: Dual certificate $\eta_{\lambda,\zeta}$ on $\mathbb S^1$. The dashed circles (orange and green) show the locations where $\eta_{\lambda,\zeta}=\pm 1$. As expected by the optimality conditions, the dual certificate is always between $-1$ and $+1$. The points where $\eta_\lambda = 1$ (resp. $\eta_\lambda = -1$) correspond exactly to the location of positive (resp. negative) Dirac deltas in the TV-regularized solution \ref{['eq:fin']}.
• Figure 4: Size of the support depending on the regularization parameter. For $\lambda$ in the interval ($\lambda_{\max}=1$, $\lambda_{\min}=5\cdot10^{-7}$, $11$ values), we compute the cardinality of the support of the solution. As expected, the number of Dirac deltas increases with the decrease of the regularization parameter $\lambda$. The maximum number of Dirac deltas is, however, still below the theoretical bound determined by the number of regions: $R(X) = 10$.
• Figure 5: Unperturbed solution and dual certificate for $\lambda = 0.2$.Left: Dual region decomposition of $\mathbb{S}^1$ and recovered atoms $\{w_*^i\}_{i=1}^2$ of the solution $\mu_{\lambda,\zeta} = \sum_{i=1}^2 c_*^i\,\delta_{w_*^i}$ with $\zeta = 0$. Both atoms lie in the interior of a dual region. Right: Corresponding dual certificate $\eta_{\lambda,\zeta}$ on $\mathbb{S}^1$. As in Figure \ref{['fig:dual_cert']}, it satisfies $|\eta_{\lambda,\zeta}| \leqslant 1$ everywhere and attains $\pm 1$ exactly at the support of $\mu_{\lambda,\zeta}$.

Theorems & Definitions (41)

• Proposition 2.1
• Definition 2.2: Minimal-norm dual certificate
• Lemma 2.3
• Lemma 3.1
• proof
• Proposition 3.2: Convergence of the dual certificates
• proof
• Definition 3.3: Dual region
• Definition 3.4: Higher–codimension dual regions
• Remark 3.5: Boundary stratification