Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of gradient flow for learning convolutional neural networks

Jona-Maria Diederen, Holger Rauhut, Ulrich Terstiege

TL;DR

The paper studies gradient flow for learning linear convolutional networks (LCNs), a simplified non-convex model of CNNs, and proves convergence to a critical point of the empirical risk under mild non-degeneracy of the data. By representing convolutions as polynomials and exploiting a balancedness property, the authors use the Łojasiewicz gradient inequality to derive global convergence for a range of loss functions, including the square loss and several robust variants. The key methodological contributions include a polynomial-factorization framework that links filter coefficients to polynomial coefficients and a global convergence theorem that hinges on a growth condition bounding the final filter norm by the loss. This work extends prior results from linear fully connected networks to convolutional architectures, providing theoretical insight into the optimization landscape of LCNs and informing the broader understanding of non-convex gradient-based learning. The findings have potential implications for the theoretical justification of gradient-based training dynamics in structured, translation-equivariant architectures.

Abstract

Convolutional neural networks are widely used in imaging and image recognition. Learning such networks from training data leads to the minimization of a non-convex function. This makes the analysis of standard optimization methods such as variants of (stochastic) gradient descent challenging. In this article we study the simplified setting of linear convolutional networks. We show that the gradient flow (to be interpreted as an abstraction of gradient descent) applied to the empirical risk defined via certain loss functions including the square loss always converges to a critical point, under a mild condition on the training data.

Convergence of gradient flow for learning convolutional neural networks

TL;DR

The paper studies gradient flow for learning linear convolutional networks (LCNs), a simplified non-convex model of CNNs, and proves convergence to a critical point of the empirical risk under mild non-degeneracy of the data. By representing convolutions as polynomials and exploiting a balancedness property, the authors use the Łojasiewicz gradient inequality to derive global convergence for a range of loss functions, including the square loss and several robust variants. The key methodological contributions include a polynomial-factorization framework that links filter coefficients to polynomial coefficients and a global convergence theorem that hinges on a growth condition bounding the final filter norm by the loss. This work extends prior results from linear fully connected networks to convolutional architectures, providing theoretical insight into the optimization landscape of LCNs and informing the broader understanding of non-convex gradient-based learning. The findings have potential implications for the theoretical justification of gradient-based training dynamics in structured, translation-equivariant architectures.

Abstract

Convolutional neural networks are widely used in imaging and image recognition. Learning such networks from training data leads to the minimization of a non-convex function. This makes the analysis of standard optimization methods such as variants of (stochastic) gradient descent challenging. In this article we study the simplified setting of linear convolutional networks. We show that the gradient flow (to be interpreted as an abstraction of gradient descent) applied to the empirical risk defined via certain loss functions including the square loss always converges to a critical point, under a mild condition on the training data.
Paper Structure (14 sections, 13 theorems, 81 equations, 1 figure)

This paper contains 14 sections, 13 theorems, 81 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Linear Convolutional Networks
  3. Learning via gradient flow
  4. Main result
  5. Structure of the paper.
  6. Acknowledgement
  7. Convolutions and Polynomials
  8. Construction of polynomials using filter vectors
  9. Auxiliary statements about polynomials
  10. Gradient flows for training linear convolutional networks
  11. Łojasiewicz' gradient inequality
  12. Convergence of the gradient flow
  13. Loss functions with convergent gradient flow
  14. Proof of Proposition \ref{['th_CompConv']}

Key Result

Proposition 1.1

The composition of $N$ convolutions $\alpha_1,\ldots,\alpha_N$ with filter sizes $\textbf{k}=(k_1,\ldots , k_N)$ and strides $\textbf{s}=(s_1,\ldots , s_N)$ is a convolution $\alpha_v$ with stride $s_{v}$ and filter width $k_{v}$, and filter given by

Figures (1)

  • Figure 1: The figure shows the various loss functions for differences of $x-y\in [-4,4]$.

Theorems & Definitions (24)

  • Proposition 1.1: Geo
  • Theorem 1.2
  • Proposition 2.1: Geo
  • proof
  • Lemma 2.2
  • proof
  • Lemma 2.3
  • proof
  • Lemma 2.4
  • proof
  • ...and 14 more