Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Breaking the Curse of Dimensionality with Convex Neural Networks

Francis Bach

TL;DR

The paper develops a convex-optimization framework for single-hidden-layer neural networks with non-decreasing positively homogeneous activations, enabling learning from a continuum of basis functions. By introducing the variation norm $\\gamma_1$ and its associated convex geometry, it proves adaptive generalization bounds that exploit low-dimensional linear structure, and shows how high-dimensional nonlinear variable selection can arise under $\\ell_1$ input-weight penalties. It also builds a RKHS connection via $\\gamma_2$, enabling kernel-based approximations, while highlighting fundamental hardness results for the incremental FW steps and offering convex relaxations (e.g., SDP, sign-vector) that preserve approximation guarantees in some regimes. Overall, the work provides deep theoretical insight into approximation, generalization, and computational aspects of convex neural networks, with explicit rates that depend on the intrinsic data subspace, and establishes a clear trade-off between adaptivity and tractability.

Abstract

We consider neural networks with a single hidden layer and non-decreasing homogeneous activa-tion functions like the rectified linear units. By letting the number of hidden units grow unbounded and using classical non-Euclidean regularization tools on the output weights, we provide a detailed theoretical analysis of their generalization performance, with a study of both the approximation and the estimation errors. We show in particular that they are adaptive to unknown underlying linear structures, such as the dependence on the projection of the input variables onto a low-dimensional subspace. Moreover, when using sparsity-inducing norms on the input weights, we show that high-dimensional non-linear variable selection may be achieved, without any strong assumption regarding the data and with a total number of variables potentially exponential in the number of ob-servations. In addition, we provide a simple geometric interpretation to the non-convex problem of addition of a new unit, which is the core potentially hard computational element in the framework of learning from continuously many basis functions. We provide simple conditions for convex relaxations to achieve the same generalization error bounds, even when constant-factor approxi-mations cannot be found (e.g., because it is NP-hard such as for the zero-homogeneous activation function). We were not able to find strong enough convex relaxations and leave open the existence or non-existence of polynomial-time algorithms.

Breaking the Curse of Dimensionality with Convex Neural Networks

TL;DR

The paper develops a convex-optimization framework for single-hidden-layer neural networks with non-decreasing positively homogeneous activations, enabling learning from a continuum of basis functions. By introducing the variation norm and its associated convex geometry, it proves adaptive generalization bounds that exploit low-dimensional linear structure, and shows how high-dimensional nonlinear variable selection can arise under input-weight penalties. It also builds a RKHS connection via , enabling kernel-based approximations, while highlighting fundamental hardness results for the incremental FW steps and offering convex relaxations (e.g., SDP, sign-vector) that preserve approximation guarantees in some regimes. Overall, the work provides deep theoretical insight into approximation, generalization, and computational aspects of convex neural networks, with explicit rates that depend on the intrinsic data subspace, and establishes a clear trade-off between adaptivity and tractability.

Abstract

We consider neural networks with a single hidden layer and non-decreasing homogeneous activa-tion functions like the rectified linear units. By letting the number of hidden units grow unbounded and using classical non-Euclidean regularization tools on the output weights, we provide a detailed theoretical analysis of their generalization performance, with a study of both the approximation and the estimation errors. We show in particular that they are adaptive to unknown underlying linear structures, such as the dependence on the projection of the input variables onto a low-dimensional subspace. Moreover, when using sparsity-inducing norms on the input weights, we show that high-dimensional non-linear variable selection may be achieved, without any strong assumption regarding the data and with a total number of variables potentially exponential in the number of ob-servations. In addition, we provide a simple geometric interpretation to the non-convex problem of addition of a new unit, which is the core potentially hard computational element in the framework of learning from continuously many basis functions. We provide simple conditions for convex relaxations to achieve the same generalization error bounds, even when constant-factor approxi-mations cannot be found (e.g., because it is NP-hard such as for the zero-homogeneous activation function). We were not able to find strong enough convex relaxations and leave open the existence or non-existence of polynomial-time algorithms.

Paper Structure

This paper contains 89 sections, 7 theorems, 100 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Learning from continuously infinitely many basis functions
  3. Variation norm
  4. Finite number of neurons.
  5. Representation from finitely many functions
  6. Corresponding reproducing kernel Hilbert space (RKHS)
  7. Random sampling.
  8. Relationship between $\mathcal{F}_1$ and $\mathcal{F}_2$.
  9. Supervised machine learning
  10. Approximation error vs. estimation error.
  11. Incremental conditional gradient algorithms
  12. Conditional gradient algorithm.
  13. Adding a new basis function.
  14. Finitely many observations.
  15. Non-smooth loss functions.
  16. ...and 74 more sections

Key Result

Proposition 1

Let $\varepsilon \in (0,1/2)$. For any function $g$ in $\mathcal{G}_1$, there exists a measure ${\mu}$ supported on at most $r$ points in $\mathcal{V}$, so that for all $z \in \mathbb{S}^{d}$. $| g(z) - \int_{\mathbb{S}^d} (v^\top z)_+ d{\mu}(v) | \leqslant \varepsilon \gamma_1(g)$, with $r \leqsla

Figures (3)

  • Figure 1: Conditional gradient algorithm for minimizing a smooth functional $J$ on $\mathcal{F}_1^\delta = \{ f \in \mathcal{F}_1, \ \gamma_1(f) \leqslant \delta\}$: going from $f_t$ to $f_{t+1}$; see text for details.
  • Figure 3: Two zonotopes in two dimensions: (left) vectors, and (right) their Minkowski sum (represented as a polygone).
  • Figure 4: Left: two zonotopes (with their generating segments) and the segments achieving the two sides of the Haussdorf distance. Right: approximation by ellipsoids.

Theorems & Definitions (7)

  • Proposition 1: Number of units - $\alpha=1$
  • Proposition 2: Finite variation on the sphere
  • Proposition 3: Approximation of Lipschitz-continuous functions on the sphere
  • Proposition 4: Norms of linear functions on the sphere
  • Proposition 5: Finite variation
  • Proposition 6: Approximation of Lipschitz-continuous functions
  • Proposition 7: Uniform deviations