Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks

El Mehdi Achour, Kathlén Kohn, Holger Rauhut

Abstract

We study geometric properties of the gradient flow for learning deep linear convolutional networks. For linear fully connected networks, it has been shown recently that the corresponding gradient flow on parameter space can be written as a Riemannian gradient flow on function space (i.e., on the product of weight matrices) if the initialization satisfies a so-called balancedness condition. We establish that the gradient flow on parameter space for learning linear convolutional networks can be written as a Riemannian gradient flow on function space regardless of the initialization. This result holds for $D$-dimensional convolutions with $D \geq 2$, and for $D =1$ it holds if all so-called strides of the convolutions are greater than one. The corresponding Riemannian metric depends on the initialization.

The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks

Abstract

We study geometric properties of the gradient flow for learning deep linear convolutional networks. For linear fully connected networks, it has been shown recently that the corresponding gradient flow on parameter space can be written as a Riemannian gradient flow on function space (i.e., on the product of weight matrices) if the initialization satisfies a so-called balancedness condition. We establish that the gradient flow on parameter space for learning linear convolutional networks can be written as a Riemannian gradient flow on function space regardless of the initialization. This result holds for -dimensional convolutions with , and for it holds if all so-called strides of the convolutions are greater than one. The corresponding Riemannian metric depends on the initialization.

Paper Structure

This paper contains 32 sections, 12 theorems, 45 equations, 1 table.

Table of Contents

  1. INTRODUCTION
  2. Gradient flow for fully-connected and convolutional linear networks
  3. Main contributions
  4. Relation to previous works
  5. Implicit bias and optimization geometry in deep linear networks.
  6. Finite-width NTK behavior.
  7. Architectural inductive bias and identifiability in structured models.
  8. Algebra-geometric approaches to deep learning and their link to optimization.
  9. Convergence and saddle avoidance under gradient flow.
  10. CONVOLUTIONS ON ONE-DIMENSIONAL SIGNALS
  11. All strides larger than one
  12. Convergence of gradient flow
  13. Some strides equal to one
  14. CONVOLUTIONS ON HIGHER-DIMENSIONAL SIGNALS
  15. FULLY-CONNECTED NETWORKS
  16. ...and 17 more sections

Key Result

Proposition 2

Let $\delta_1, \ldots, \delta_{H-1} \in \mathbb{R}$, and consider non-zero filters $w_1 \in \mathbb{R}^{k_1}$, …, $w_H \in \mathbb{R}^{k_H}$. Then, there are $2^{H-1}$ scalar tuples $(\lambda_1, \ldots, \lambda_H) \in \mathbb{R}^H$ such that $\lambda_1 \cdots \lambda_H = 1$ and These scalar tuples are all equal up to sign changes of the components. $\blacktriangleleft$$\blacktriangleleft$

Theorems & Definitions (29)

  • Remark 1
  • Proposition 2
  • Theorem 4
  • Lemma 5
  • Corollary 6
  • Theorem 7
  • proof
  • Proposition 8
  • Lemma 9
  • Corollary 10
  • ...and 19 more