Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numerics
Kang Liu, Enrique Zuazua
TL;DR
This work addresses three non-convex sparse-learning problems for shallow neural networks by employing mean-field relaxations to convex measure-valued formulations. A Representer Theorem-based analysis shows there is no relaxation gap for $P\ge N$, enabling a reduction to $P=N$ and characterizing solutions as finite empirical measures with at most $N$ atoms. The authors derive a generalization bound based on the Kantorovich–Rubinstein distance, provide practical guidance for hyperparameter selection, and develop discretization and sparsification algorithms that connect convex relaxations to primal sparse representations. Numerical experiments illustrate the trade-offs between exact/approximate representation and regression, including a sparsification pipeline that yields strong performance with far fewer active neurons, and a double-descent interpretation in random-feature regimes. The framework unifies theory and computation for shallow NNs, offering scalable paths from infinite-dimensional relaxations to sparse, high-performing models with principled hyperparameter choices.
Abstract
In this work, we address three non-convex optimization problems associated with the training of shallow neural networks (NNs) for exact and approximate representation, as well as for regression tasks. Through a mean-field approach, we convexify these problems and, applying a representer theorem, prove the absence of relaxation gaps. We establish generalization bounds for the resulting NN solutions, assessing their predictive performance on test datasets and, analyzing the impact of key hyperparameters on these bounds, propose optimal choices. On the computational side, we examine the discretization of the convexified problems and derive convergence rates. For low-dimensional datasets, these discretized problems are efficiently solvable using the simplex method. For high-dimensional datasets, we propose a sparsification algorithm that, combined with gradient descent for over-parameterized shallow NNs, yields effective solutions to the primal problems.