Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convex Formulations for Training Two-Layer ReLU Neural Networks

Karthik Prakhya, Tolga Birdal, Alp Yurtsever

TL;DR

The paper addresses the non-convexity of training two-layer ReLU neural networks by formulating the problem as a convex copositive program, proving exact equivalence with the original problem at a finite critical width and extending to infinite width via a measure-based representation. To enable practical computation, it introduces a polynomial-time SDP relaxation that replaces the copositive constraint with a doubly non-negative cone while preserving the problem's core structure. Empirical results on synthetic and real datasets show the SDP relaxation is reasonably tight and yields competitive test accuracy compared to Neural Network Gaussian Processes and Neural Tangent Kernels, with a rounding step enabling extraction of trained weights. The work provides a principled lifting-based, width-agnostic framework for neural network training, offering insights into the role of width, the potential of convex relaxations, and avenues for future quantum or rank-bound approaches to copositive programs.

Abstract

Solving non-convex, NP-hard optimization problems is crucial for training machine learning models, including neural networks. However, non-convexity often leads to black-box machine learning models with unclear inner workings. While convex formulations have been used for verifying neural network robustness, their application to training neural networks remains less explored. In response to this challenge, we reformulate the problem of training infinite-width two-layer ReLU networks as a convex completely positive program in a finite-dimensional (lifted) space. Despite the convexity, solving this problem remains NP-hard due to the complete positivity constraint. To overcome this challenge, we introduce a semidefinite relaxation that can be solved in polynomial time. We then experimentally evaluate the tightness of this relaxation, demonstrating its competitive performance in test accuracy across a range of classification tasks.

Convex Formulations for Training Two-Layer ReLU Neural Networks

TL;DR

The paper addresses the non-convexity of training two-layer ReLU neural networks by formulating the problem as a convex copositive program, proving exact equivalence with the original problem at a finite critical width and extending to infinite width via a measure-based representation. To enable practical computation, it introduces a polynomial-time SDP relaxation that replaces the copositive constraint with a doubly non-negative cone while preserving the problem's core structure. Empirical results on synthetic and real datasets show the SDP relaxation is reasonably tight and yields competitive test accuracy compared to Neural Network Gaussian Processes and Neural Tangent Kernels, with a rounding step enabling extraction of trained weights. The work provides a principled lifting-based, width-agnostic framework for neural network training, offering insights into the role of width, the potential of convex relaxations, and avenues for future quantum or rank-bound approaches to copositive programs.

Abstract

Solving non-convex, NP-hard optimization problems is crucial for training machine learning models, including neural networks. However, non-convexity often leads to black-box machine learning models with unclear inner workings. While convex formulations have been used for verifying neural network robustness, their application to training neural networks remains less explored. In response to this challenge, we reformulate the problem of training infinite-width two-layer ReLU networks as a convex completely positive program in a finite-dimensional (lifted) space. Despite the convexity, solving this problem remains NP-hard due to the complete positivity constraint. To overcome this challenge, we introduce a semidefinite relaxation that can be solved in polynomial time. We then experimentally evaluate the tightness of this relaxation, demonstrating its competitive performance in test accuracy across a range of classification tasks.

Paper Structure

This paper contains 26 sections, 8 theorems, 45 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Semidefinite programming
  4. CP Formulation for Training Two-Layer ReLU Networks
  5. The Infinite Width Regime
  6. SDP Relaxation for Training Two-Layer ReLU Networks
  7. Numerical Experiments
  8. Datasets
  9. Computational environment & complexity
  10. Empirical Approximation Ratio of the SDP Relaxation
  11. Prediction Performance of the SDP Relaxation
  12. Related Works
  13. Convex optimization for neural network training
  14. Infinite-Width Neural Networks (IWNNs)
  15. Other related works
  16. ...and 11 more sections

Key Result

Lemma 1

Let $\bm{W}$ be a completely positive matrix of rank $r \geq 2$. Then, the CP-rank of $\bm{W}$ cannot exceed ${r(r+1)}/{2} - 1$. As an immediate corollary, the CP-rank of any $n \times n$ completely positive matrix for $n \geq 2$ is bounded above by ${n(n+1)}/{2}-1$, which we denote as the maximal C

Figures (2)

  • Figure 1: Cones.
  • Figure 2: Objective value (training loss) for Random and Spiral datasets trained with SGD using different numbers of hidden neurons. The dashed line indicates the lower bound obtained from solving SDP-NN with the interior point method as a reference.

Theorems & Definitions (28)

  • Definition 1: Positive semidefinite cone
  • Definition 2: Copositive cone
  • Definition 3: Completely positive cone
  • Definition 4: Non-negative cone
  • Definition 5: Doubly non-negative cone
  • Definition 6: Rank
  • Definition 7: CP-rank
  • Lemma 1
  • Definition 8: Two-Layer ReLU Network
  • Theorem 1: CP Formulation
  • ...and 18 more