Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embedding principle of homogeneous neural network for classification problem

Jiahan Zhang, Yaoyu Zhang, Tao Luo

TL;DR

This work addresses how margin-maximizing, KKT-aligned solutions in homogeneous networks relate when width is increased via embedding transformations. It introduces the KKT Point Embedding Principle, proving that a linear isometry $T$ maps KKT points from the smaller network's $P_\Phi$ to the larger network's $P_{\tilde{\Phi}}$ and preserves gradient-flow dynamics, including $\boldsymbol{\eta}(t) = T\boldsymbol{\theta}(t)$ and the mapping of $\omega$-limit sets. The analysis provides static proofs for neuron splitting in fully-connected networks and channel splitting in CNNs, and demonstrates that the dynamic trajectories and limit directions are preserved, supported by experiments on 2D toy data and MNIST. Overall, the results offer a theoretical scaffold for width overparameterization, parameter redundancy, and the structure of margin-aligned solutions across architectures, with implications for implicit regularization and transfer of solutions.

Abstract

In this paper, we study the Karush-Kuhn-Tucker (KKT) points of the associated maximum-margin problem in homogeneous neural networks, including fully-connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths generated. We introduce and formalize the \textbf{KKT point embedding principle}, establishing that KKT points of a homogeneous network's max-margin problem ($P_Φ$) can be embedded into the KKT points of a larger network's problem ($P_{\tildeΦ}$) via specific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, we connect this static embedding to the dynamics of gradient flow training with smooth losses. We demonstrate that trajectories initiated from appropriately mapped points remain mapped throughout training and that the resulting $ω$-limit sets of directions are correspondingly mapped, thereby preserving the alignment with KKT directions dynamically when directional convergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimization in homogeneous networks of varying sizes.

Embedding principle of homogeneous neural network for classification problem

TL;DR

This work addresses how margin-maximizing, KKT-aligned solutions in homogeneous networks relate when width is increased via embedding transformations. It introduces the KKT Point Embedding Principle, proving that a linear isometry maps KKT points from the smaller network's to the larger network's and preserves gradient-flow dynamics, including and the mapping of -limit sets. The analysis provides static proofs for neuron splitting in fully-connected networks and channel splitting in CNNs, and demonstrates that the dynamic trajectories and limit directions are preserved, supported by experiments on 2D toy data and MNIST. Overall, the results offer a theoretical scaffold for width overparameterization, parameter redundancy, and the structure of margin-aligned solutions across architectures, with implications for implicit regularization and transfer of solutions.

Abstract

In this paper, we study the Karush-Kuhn-Tucker (KKT) points of the associated maximum-margin problem in homogeneous neural networks, including fully-connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths generated. We introduce and formalize the \textbf{KKT point embedding principle}, establishing that KKT points of a homogeneous network's max-margin problem () can be embedded into the KKT points of a larger network's problem () via specific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, we connect this static embedding to the dynamics of gradient flow training with smooth losses. We demonstrate that trajectories initiated from appropriately mapped points remain mapped throughout training and that the resulting -limit sets of directions are correspondingly mapped, thereby preserving the alignment with KKT directions dynamically when directional convergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimization in homogeneous networks of varying sizes.
Paper Structure (46 sections, 9 theorems, 15 equations, 7 figures, 2 tables)

This paper contains 46 sections, 9 theorems, 15 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Optimization dynamics and implicit bias
  4. Network embedding principles
  5. KKT conditions in optimization and machine learning
  6. Preliminaries
  7. Basic notations
  8. Binary classification setup
  9. Gradient flow
  10. The KKT point embedding principle: Static setting
  11. The KKT point embedding principle: a general framework
  12. Applications: KKT embedding via neuron and channel splitting
  13. Neuron splitting in fully-connected networks
  14. Channel splitting in convolutional neural networks
  15. Connection to training dynamics via gradient flow
  16. ...and 31 more sections

Key Result

Theorem 4.2

Let $f, g_k : \mathbb{R}^m \to \mathbb{R}$ and $\tilde{f}, \tilde{g}_{k} : \mathbb{R}^{\tilde{m}} \to \mathbb{R}$ be locally Lipschitz. Consider problems: Let $T: \mathbb{R}^{m} \to \mathbb{R}^{\tilde{m}}$ be linear. Suppose: If $\boldsymbol{\theta}^{*}$ is a KKT point of ($P$), then $\boldsymbol{\eta}^{*} = T \boldsymbol{\theta}^{*}$ is a KKT point of ($\tilde{P}$).

Figures (7)

  • Figure 1: Overview of Theoretical Contributions. This figure illustrates the logical flow of our paper's theoretical framework. The analysis begins with the static principles (top section), which are then shown to apply to specific network architectures. These static results form the basis for the dynamic analysis (bottom row), which shows the preservation of training trajectories and their limit sets.
  • Figure 2: Empirical validation using MLP with GD on 2D toy data (Exp. 1). (Left) The trajectory error remains near machine precision throughout training. (Right) The training loss converges towards zero, indicating successful training.
  • Figure 3: Trajectory error comparison for MLP experiments using Gradient Descent (GD). (Left) Exp. 1 on separable data shows error near machine precision. (Right) Exp. 2 on non-separable data also shows error remaining near machine precision, demonstrating robustness to data conditions.
  • Figure 4: Trajectory error comparison for MLP experiments using Stochastic Gradient Descent (SGD). (Left) Exp. 3, using identical mini-batches for both networks, maintains error near machine precision (moving average shown). (Right) Exp. 4, using different mini-batches, shows substantial error growth, highlighting the necessity of identical data sequences.
  • Figure 5: Results for the CNN experiment using GD on separable data (Exp. 5). (Left) The trajectory error remains small ($<10^{-9}$) but exhibits a slight upward trend, consistent with accumulated precision errors. (Right) The training loss converges successfully.
  • ...and 2 more figures

Theorems & Definitions (30)

  • Definition 3.1: Homogeneous neural network
  • Definition 3.2: Margin
  • Definition 3.3: KKT conditions for non-smooth problems
  • Definition 3.4: Minimum-norm max-margin problem $P_{\Phi}$
  • Theorem 4.2: General KKT mapping via linear transformation
  • proof
  • Definition 4.3: KKT point preserving transformation
  • Proposition 4.4: Composition of KKT point preserving transformations
  • proof
  • Theorem 4.5: Equivalence conditions for KKT embedding
  • ...and 20 more