Generalization Bounds for Rank-sparse Neural Networks

TL;DR

The paper investigates how approximate low-rank structure in neural network weights, captured via Schatten $p$-quasi-norms, influences generalization. By introducing a parametric interpolation between norm-based and rank-based bounds, it derives generalization guarantees for linear, fully-connected, and CNN architectures that adapt to the degree of rank sparsity across layers. For small $p$, the bounds reveal a rank-aware sample complexity that scales with network width $W$, depth $L$, and layer rank $r$, while increasing $p$ recovers traditional norm-based bounds; a Loss Augmentation variant further tightens the bounds using intermediary activations. These results offer a principled explanation for empirically observed low-rank structures in deep networks and suggest practical bounds that adapt to the implicit dimensionality of learned representations, with extensions to CNNs and potential applicability to modern architectures.

Abstract

It has been recently observed in much of the literature that neural networks exhibit a bottleneck rank property: for larger depths, the activation and weights of neural networks trained with gradient-based methods tend to be of approximately low rank. In fact, the rank of the activations of each layer converges to a fixed value referred to as the ``bottleneck rank'', which is the minimum rank required to represent the training data. This perspective is in line with the observation that regularizing linear networks (without activations) with weight decay is equivalent to minimizing the Schatten $p$ quasi norm of the neural network. In this paper we investigate the implications of this phenomenon for generalization. More specifically, we prove generalization bounds for neural networks which exploit the approximate low rank structure of the weight matrices if present. The final results rely on the Schatten $p$ quasi norms of the weight matrices: for small $p$, the bounds exhibit a sample complexity $ \widetilde{O}(WrL^2)$ where $W$ and $L$ are the width and depth of the neural network respectively and where $r$ is the rank of the weight matrices. As $p$ increases, the bound behaves more like a norm-based bound instead.

Figures (6)

• Figure 1: Empirical comparison between our generalization results and previous works.
• Figure 2: Spectral Decay of Intermediary Layers in the MNIST Dataset (Fully Connected).
• Figure 3: Numerical Comparison of our Bounds with other Bounds from the Literature (CNNs, CIFAR-10).
• Figure 4: Illustration of the Spectral Decay of the Trained Models on the Layer fc2.
• Figure 5: Chosen Values of $\pl$ (with Loss Function Augmentation) for the most Overparametrized CNN Model (width $=1000$)

Theorems & Definitions (65)

• theorem 1: cf. dai2021representation, Theorem 1
• theorem 2
• theorem 3
• theorem 4: Cf. Theorem \ref{['thm:posthoclaug']}
• proposition 1: Simplified form of Proposition \ref{['prop:Linfinity_Schatten_cover']}: covering number bound for one layer
• proof : Proof Sketch
• proposition 2: Simplified form of Proposition \ref{['prop:genboundfixedclass']}
• proof : Proof Sketch
• theorem 5
• theorem 6: Cf. Theorem \ref{['thm:posthocconvlaug']}