Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep-ICE: The first globally optimal algorithm for empirical risk minimization of two-layer maxout and ReLU networks

Xi He, Yi Miao, Max A. Little

TL;DR

This work tackles the problem of exactly training two-layer neural networks with discrete $0$-$1$ loss by introducing Deep-ICE, a globally optimal algorithm with worst-case complexity $O\left(N^{DK+1}\right)$ for fixed input dimension $D$ and hidden units $K$. It builds a theoretical framework around list-based algebras and fusion laws to enable a recursive, fusion-friendly search over $K$-hyperplane configurations and $2^K$ orientations, avoiding pre-storing all hyperplanes and exploiting symmetry to reduce computation. A key technical contribution is the nested combination generator (nestedCombs) on join-lists, which allows memory-efficient, GPU-friendly exhaustive exploration and fusable components (min$_{0-1}$, eval, cp). To scale to larger datasets, the authors introduce a coreset selection method that preserves optimal configurations while dramatically reducing data size, yielding 20–30% fewer misclassifications than SVMs and gradient-based maxout baselines on multiple UCI datasets. Empirically, Deep-ICE achieves exact or near-exact solutions on small problems and substantial performance gains on larger data when combined with coresets, highlighting its potential for exact ERM in interpretable two-layer architectures and suggesting broader applicability to nested combinatorial optimization problems.

Abstract

This paper introduces the first globally optimal algorithm for the empirical risk minimization problem of two-layer maxout and ReLU networks, i.e., minimizing the number of misclassifications. The algorithm has a worst-case time complexity of $O\left(N^{DK+1}\right)$, where $K$ denotes the number of hidden neurons and $D$ represents the number of features. It can be can be generalized to accommodate arbitrary computable loss functions without affecting its computational complexity. Our experiments demonstrate that the proposed algorithm provides provably exact solutions for small-scale datasets. To handle larger datasets, we introduce a novel coreset selection method that reduces the data size to a manageable scale, making it feasible for our algorithm. This extension enables efficient processing of large-scale datasets and achieves significantly improved performance, with a 20-30\% reduction in misclassifications for both training and prediction, compared to state-of-the-art approaches (neural networks trained using gradient descent and support vector machines), when applied to the same models (two-layer networks with fixed hidden nodes and linear models).

Deep-ICE: The first globally optimal algorithm for empirical risk minimization of two-layer maxout and ReLU networks

TL;DR

This work tackles the problem of exactly training two-layer neural networks with discrete - loss by introducing Deep-ICE, a globally optimal algorithm with worst-case complexity for fixed input dimension and hidden units . It builds a theoretical framework around list-based algebras and fusion laws to enable a recursive, fusion-friendly search over -hyperplane configurations and orientations, avoiding pre-storing all hyperplanes and exploiting symmetry to reduce computation. A key technical contribution is the nested combination generator (nestedCombs) on join-lists, which allows memory-efficient, GPU-friendly exhaustive exploration and fusable components (min, eval, cp). To scale to larger datasets, the authors introduce a coreset selection method that preserves optimal configurations while dramatically reducing data size, yielding 20–30% fewer misclassifications than SVMs and gradient-based maxout baselines on multiple UCI datasets. Empirically, Deep-ICE achieves exact or near-exact solutions on small problems and substantial performance gains on larger data when combined with coresets, highlighting its potential for exact ERM in interpretable two-layer architectures and suggesting broader applicability to nested combinatorial optimization problems.

Abstract

This paper introduces the first globally optimal algorithm for the empirical risk minimization problem of two-layer maxout and ReLU networks, i.e., minimizing the number of misclassifications. The algorithm has a worst-case time complexity of , where denotes the number of hidden neurons and represents the number of features. It can be can be generalized to accommodate arbitrary computable loss functions without affecting its computational complexity. Our experiments demonstrate that the proposed algorithm provides provably exact solutions for small-scale datasets. To handle larger datasets, we introduce a novel coreset selection method that reduces the data size to a manageable scale, making it feasible for our algorithm. This extension enables efficient processing of large-scale datasets and achieves significantly improved performance, with a 20-30\% reduction in misclassifications for both training and prediction, compared to state-of-the-art approaches (neural networks trained using gradient descent and support vector machines), when applied to the same models (two-layer networks with fixed hidden nodes and linear models).
Paper Structure (23 sections, 6 theorems, 28 equations, 1 figure, 2 tables, 2 algorithms)

This paper contains 23 sections, 6 theorems, 28 equations, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Theory
  3. Theory of lists
  4. List homomorphisms
  5. Fusion laws
  6. Problem specification
  7. An exhaustive search specification
  8. An efficient nested combination generator join-list
  9. Deep incremental cell enumeration (Deep-ICE) algorithm and symmetry fusion
  10. $\mathit{DeepICE}\left(D,K\right)=\mathit{min}_{\text{0-1}}\left(K\right)\circ\mathit{eval}\left(K\right)\circ\mathit{cp}\left(\mathit{basgns}\left(K\right)\right)\circ\mathit{\mathit{nestedCombs}}\left(D,K\right)\Rightarrow\{\text{symmetry fusion}\}=\mathit{min}_{\text{0-1}}\left(K\right)\circ\mathit{eval}\left(K\right)\circ\mathit{cp}\left(\mathit{basgns}\left(K-1\right)\right)\circ\mathit{\mathit{nestedCombs}}\left(D,K\right)\Rightarrow\{\mathit{eval}^{\prime}\left(K\right)=\mathit{eval}\left(K\right)\circ\mathit{\mathit{\mathit{c}p}}\left(\mathit{basgns}\left(K\right)\right)\}=\mathit{min}_{\text{0-1}}\left(K\right)\circ\mathit{eval}^{\prime}\left(K-1\right)\circ\mathit{\mathit{nestedCombs}}\left(D,K\right).$
  11. Empirical analysis
  12. Exact solution vs. gradient-based training on a real-world dataset
  13. Exact solution over coresets
  14. Discussion and conclusion
  15. Appendix
  16. ...and 8 more sections

Key Result

Theorem 1

Fusion law for the cons-list. Let $f$ be a function and let $h$ and $g$ be two cons-list homomorphisms defined by the algebras $alg$ and $alg^{\prime}$, respectively. The fusion law states that $f\circ h=g$if the fusion condition $f\left(alg\left(a,h\left(x\right)\right)\right)=alg^{\prime}\left(a,h

Figures (1)

  • Figure 1: The global optimal solution of a rank-2 maxout network with one neuron on a real-world dataset containing $N=704$ data items in $\mathbb{R}^{2}$. (a) Exact solution returned by the Deep-ICE algorithm. (b) Solution from training the same network using gradient descent.

Theorems & Definitions (10)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • proof
  • Lemma 1
  • proof
  • Theorem 4
  • proof
  • Lemma 2
  • proof