Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Block Coordinate Descent for Neural Networks Provably Finds Global Minima

Shunta Akiyama

TL;DR

This paper studies block coordinate descent (BCD) as an optimization framework for training deep neural networks and provides the first global convergence guarantee to arbitrarily small training loss for networks with an arbitrary number of layers under strictly increasing activations. It introduces auxiliary representations to decouple layer updates, proves exponential-like reduction of the output-layer loss while keeping hidden-layer losses controlled, and establishes a Rademacher-based generalization bound via spectral-norm control. To accommodate ReLU activations, the authors design a ReLU-specific BCD with skip connections and non-negative projections, enabling global convergence in a practical setting. Theoretical results are complemented by synthetic experiments showing monotone-activation and ReLU networks converge to small training loss, with SVB improving stability and performance, and skip connections proving crucial for ReLU. Overall, the work advances understanding of optimization dynamics beyond the NTK regime and provides concrete, testable guarantees for deep-BACD training strategies and their generalization behavior.

Abstract

In this paper, we consider a block coordinate descent (BCD) algorithm for training deep neural networks and provide a new global convergence guarantee under strictly monotonically increasing activation functions. While existing works demonstrate convergence to stationary points for BCD in neural networks, our contribution is the first to prove convergence to global minima, ensuring arbitrarily small loss. We show that the loss with respect to the output layer decreases exponentially while the loss with respect to the hidden layers remains well-controlled. Additionally, we derive generalization bounds using the Rademacher complexity framework, demonstrating that BCD not only achieves strong optimization guarantees but also provides favorable generalization performance. Moreover, we propose a modified BCD algorithm with skip connections and non-negative projection, extending our convergence guarantees to ReLU activation, which are not strictly monotonic. Empirical experiments confirm our theoretical findings, showing that the BCD algorithm achieves a small loss for strictly monotonic and ReLU activations.

Block Coordinate Descent for Neural Networks Provably Finds Global Minima

TL;DR

This paper studies block coordinate descent (BCD) as an optimization framework for training deep neural networks and provides the first global convergence guarantee to arbitrarily small training loss for networks with an arbitrary number of layers under strictly increasing activations. It introduces auxiliary representations to decouple layer updates, proves exponential-like reduction of the output-layer loss while keeping hidden-layer losses controlled, and establishes a Rademacher-based generalization bound via spectral-norm control. To accommodate ReLU activations, the authors design a ReLU-specific BCD with skip connections and non-negative projections, enabling global convergence in a practical setting. Theoretical results are complemented by synthetic experiments showing monotone-activation and ReLU networks converge to small training loss, with SVB improving stability and performance, and skip connections proving crucial for ReLU. Overall, the work advances understanding of optimization dynamics beyond the NTK regime and provides concrete, testable guarantees for deep-BACD training strategies and their generalization behavior.

Abstract

In this paper, we consider a block coordinate descent (BCD) algorithm for training deep neural networks and provide a new global convergence guarantee under strictly monotonically increasing activation functions. While existing works demonstrate convergence to stationary points for BCD in neural networks, our contribution is the first to prove convergence to global minima, ensuring arbitrarily small loss. We show that the loss with respect to the output layer decreases exponentially while the loss with respect to the hidden layers remains well-controlled. Additionally, we derive generalization bounds using the Rademacher complexity framework, demonstrating that BCD not only achieves strong optimization guarantees but also provides favorable generalization performance. Moreover, we propose a modified BCD algorithm with skip connections and non-negative projection, extending our convergence guarantees to ReLU activation, which are not strictly monotonic. Empirical experiments confirm our theoretical findings, showing that the BCD algorithm achieves a small loss for strictly monotonic and ReLU activations.
Paper Structure (51 sections, 27 theorems, 130 equations, 4 figures, 3 algorithms)

This paper contains 51 sections, 27 theorems, 130 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Convergence Guarantees for Gradient Descent and Stochastic Gradient Descent in Neural Networks
  4. Generalization Error Bounds for Multi-Layer Neural Networks
  5. Preliminaries
  6. Notations
  7. Problem Settings
  8. Block Coordinate Descent
  9. Initialization:
  10. Update of V:
  11. Update of W:
  12. Global Convergence of Block Coordinate Descent
  13. Global Convergence with Monotonically Increasing Activation
  14. Generalization Error Bound
  15. ReLU Activation
  16. ...and 36 more sections

Key Result

Theorem 5.1

Suppose that activation $\sigma$ satisfies assu:activation. Let $s\coloneqq \sigma_{\min}(X)$ denote the smallest singular value of the data matrix $X$. Let $R_i = |W_LV_{L-1,i}-y_i|$ be the initial residual at the output layer, and define $R\coloneqq\sum_{i=1}^nR_i^2$, $R_{\max}\coloneqq \underset{ where $\mathbf{W}=(W_1,\dots,W_L)$ and $\mathbf{V}=(V_{1,1},\dots,V_{L-1,n})$ are the output of alg

Figures (4)

  • Figure 1: Graphical comparison between backpropagation and block coordinate descent. In contrast, the block coordinate descent approach introduces auxiliary variables $V_{j,i}$, which serve as approximations of the hidden layer outputs, enabling layer-wise updates and a decoupled optimization structure (see \ref{['sec:BCD']} for details).
  • Figure 2: Training loss of \ref{['algo:bcd']} with LeakyReLU activation ($\alpha=0.5$).
  • Figure 3: Training loss of \ref{['algo:bcdReLU']} with ReLU activation.
  • Figure 4: Training loss curves for networks of depth $L=4$, $L=8$, and $L=12$.

Theorems & Definitions (53)

  • Remark 4.1
  • Remark 4.2
  • Remark 4.3
  • Theorem 5.1
  • Remark 5.2
  • Definition 5.4: Generalization gap
  • Theorem 5.5
  • Lemma 6.1
  • Lemma 6.2
  • Theorem 6.3
  • ...and 43 more