Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Implicit Regularization Towards Rank Minimization in ReLU Networks

Nadav Timor, Gal Vardi, Ohad Shamir

TL;DR

This paper examines whether gradient-flow optimization in nonlinear ReLU networks implicitly regularizes toward low-rank weight matrices. It reveals a nuanced picture: GF can fail to minimize rank even for simple depth-2, width-2 architectures on tiny, realizable datasets, yet for sufficiently deep networks GF tends to produce low-rank solutions under both square loss with norm minimization and exponentially-tailed losses with margin-maximization. The authors establish negative results via geometric GF dynamics on a two-input, two-output setting, and positive results by constructing deep, balanced representations and proving norm-rank relations that imply low-rank structure in the limit. The findings highlight a strong dependence on architecture and loss regime for implicit rank regularization, with practical implications for understanding generalization and compression in deep networks.

Abstract

We study the conjectured relationship between the implicit regularization in neural networks, trained with gradient-based methods, and rank minimization of their weight matrices. Previously, it was proved that for linear networks (of depth 2 and vector-valued outputs), gradient flow (GF) w.r.t. the square loss acts as a rank minimization heuristic. However, understanding to what extent this generalizes to nonlinear networks is an open problem. In this paper, we focus on nonlinear ReLU networks, providing several new positive and negative results. On the negative side, we prove (and demonstrate empirically) that, unlike the linear case, GF on ReLU networks may no longer tend to minimize ranks, in a rather strong sense (even approximately, for "most" datasets of size 2). On the positive side, we reveal that ReLU networks of sufficient depth are provably biased towards low-rank solutions in several reasonable settings.

Implicit Regularization Towards Rank Minimization in ReLU Networks

TL;DR

This paper examines whether gradient-flow optimization in nonlinear ReLU networks implicitly regularizes toward low-rank weight matrices. It reveals a nuanced picture: GF can fail to minimize rank even for simple depth-2, width-2 architectures on tiny, realizable datasets, yet for sufficiently deep networks GF tends to produce low-rank solutions under both square loss with norm minimization and exponentially-tailed losses with margin-maximization. The authors establish negative results via geometric GF dynamics on a two-input, two-output setting, and positive results by constructing deep, balanced representations and proving norm-rank relations that imply low-rank structure in the limit. The findings highlight a strong dependence on architecture and loss regime for implicit rank regularization, with practical implications for understanding generalization and compression in deep networks.

Abstract

We study the conjectured relationship between the implicit regularization in neural networks, trained with gradient-based methods, and rank minimization of their weight matrices. Previously, it was proved that for linear networks (of depth 2 and vector-valued outputs), gradient flow (GF) w.r.t. the square loss acts as a rank minimization heuristic. However, understanding to what extent this generalizes to nonlinear networks is an open problem. In this paper, we focus on nonlinear ReLU networks, providing several new positive and negative results. On the negative side, we prove (and demonstrate empirically) that, unlike the linear case, GF on ReLU networks may no longer tend to minimize ranks, in a rather strong sense (even approximately, for "most" datasets of size 2). On the positive side, we reveal that ReLU networks of sufficient depth are provably biased towards low-rank solutions in several reasonable settings.
Paper Structure (18 sections, 20 theorems, 97 equations, 3 figures)

This paper contains 18 sections, 20 theorems, 97 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations.
  4. Neural networks.
  5. Optimization problem and gradient flow (GF).
  6. Gradient flow does not even approximately minimize ranks
  7. An empirical result
  8. Rank minimization in deep networks with small $\ell_2$ norm
  9. Rank minimization in deep networks with exponentially-tailed losses
  10. Proof ideas
  11. Theorem \ref{['T2']}
  12. Theorem \ref{['T3']}
  13. Theorems \ref{['thm:positive-square-loss']} and \ref{['thm:positive-exp']}
  14. Proof of Thm. \ref{['T1']}
  15. Proof of Thm. \ref{['T2']}
  16. ...and 3 more sections

Key Result

Theorem 1

Given any labeled dataset $(X, Y) \in \mathbb{R}^{2 \times 2} \times \mathbb{R}^{2 \times 2}$ of two inputs $\mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^2$ with a strictly positive angle between them, i.e., $\measuredangle(\mathbf{x}_1, \mathbf{x}_2) > 0$, there exists a zero-loss solution $N_{W,V}$ w

Figures (3)

  • Figure 1: A histogram of the stable (numerical) ranks at convergence. In all runs, we converge to networks with stable ranks which seem bounded away from $1$. Namely, gradient descent does not even approximately minimize the ranks.
  • Figure 2: Regions $\mathcal{D}$, $\mathcal{S}_1$, $\mathcal{S}_2$, $\mathcal{S}$.
  • Figure 3: Regions ${\cal F}_i$ (in green) and ${\cal A}_i$ (the union of the green and orange regions). A dashed line marks an open boundary.

Theorems & Definitions (34)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1: Paraphrased from lyu2019gradient and ji2020directional
  • Theorem 5
  • Lemma 2
  • proof
  • Definition 1
  • Lemma 3
  • ...and 24 more