An Overview of Low-Rank Structures in the Training and Adaptation of Large Models

TL;DR

This work investigates how low-rank structures naturally arise during the training and adaptation of large models, providing both theoretical and practical avenues to reduce computation. It couples analyses of gradient dynamics in deep linear networks with practical techniques like LoRA and various adaptive low-rank training schemes to show how updates concentrate in low-dimensional subspaces and how this can be exploited for parameter-efficient fine-tuning. The review highlights two complementary viewpoints—structure along the gradient trajectory and implicit structure at convergence via regularization—and demonstrates through both theory and empirical findings that low-rank methods can maintain performance while drastically reducing memory and compute. The findings have significant implications for scaling up training and inference in large models, including LLMs and vision-language systems, and point to open questions in extending these insights to nonlinear architectures and broader masking strategies.

Abstract

The rise of deep learning has revolutionized data processing and prediction in signal processing and machine learning, yet the substantial computational demands of training and deploying modern large-scale deep models present significant challenges, including high computational costs and energy consumption. Recent research has uncovered a widespread phenomenon in deep networks: the emergence of low-rank structures in weight matrices and learned representations during training. These implicit low-dimensional patterns provide valuable insights for improving the efficiency of training and fine-tuning large-scale models. Practical techniques inspired by this phenomenon, such as low-rank adaptation (LoRA) and training, enable significant reductions in computational cost while preserving model performance. In this paper, we present a comprehensive review of recent advances in exploiting low-rank structures for deep learning and shed light on their mathematical foundations. Mathematically, we present two complementary perspectives on understanding the low-rankness in deep networks: (i) the emergence of low-rank structures throughout the whole optimization dynamics of gradient and (ii) the implicit regularization effects that induce such low-rank structures at convergence. From a practical standpoint, studying the low-rank learning dynamics of gradient descent offers a mathematical foundation for understanding the effectiveness of LoRA in fine-tuning large-scale models and inspires parameter-efficient low-rank training strategies. Furthermore, the implicit low-rank regularization effect helps explain the success of various masked training approaches in deep neural networks, ranging from dropout to masked self-supervised learning.

Figures (10)

• Figure 1: Prevalence of low-rank weight updates in various deep networks. Each plot visualizes the singular values of the weight updates from initialization for the penultimate layer weight matrix for different types of network architectures: deep linear network (DLN), multi-layer perception (MLP), VGG, and ViT-B. The linear network is trained on MNIST with mean square error loss. The MLP is trained on MNIST with cross-entropy loss. The VGG and ViT-B networks are both trained on CIFAR-10 with cross-entropy loss. The result shows a prevalent phenomenon across linear and nonlinear networks -- gradient descent only updates a small portion of the singular values, while the others remain small and almost unchanged. Figure courtesy of kwon2023efficient.
• Figure 2: Singular values of activations of the last layer in deep architectures trained with dropout rate $60\%$, as a function of training epoch. Deep Linear Network (DLN) and Multi-Layer Perceptron (MLP) are trained on synthetic data with MSE loss, while ResNet is trained on CIFAR-10 dataset of natural images with cross-entropy loss. Notably, the activations of the layer gradually become low-rank as masked training proceeds.
• Figure 3: Evolution of SVD of weight matrices for deep matrix factorization. We visualize the SVD dynamics of the first layer weight matrix of an $L=3$ layer deep matrix factorization for a random matrix with $d = 30$, $r=3$, $\epsilon_l = 1$ throughout GD without weight decay ($\lambda = 0$). Left: Magnitude of the $i$-th singular value $\sigma_i(t)$ at iteration $t$. Middle: Angle $\angle(\bm v_i(t), \bm v_i(0))$ between the $i$-th right singular vector at iteration $t$ and initialization. Right: Angle $\angle(\bm u_i(t), \bm u_i(0))$ between the $i$-th left singular vector at iteration $t$ and initialization.
• Figure 4: Evolution of SVD of weight matrices for deep low-rank adaptation. We visualize the SVD dynamics of an $L=3$ layer deep matrix factorization's end-to-end product employed for fine-tuning the 11th layer value matrix in BERT, with $d = 768$, $\epsilon_l = 1$ throughout Adam. Left: Magnitude of the $i$-th singular value $\sigma_i(t)$ at iteration $t$. Middle: Angle $\angle(\bm v_i(t), \bm v_i(0))$ between the $i$-th right singular vector at iteration $t$ and initialization. Right: Angle $\angle(\bm u_i(t), \bm u_i(0))$ between the $i$-th left singular vector at iteration $t$ and initialization.
• Figure 5: Highlighting the inefficiency of LoRA that arises from using asymmetric initializations on BERT. Existing papers explore ways to balance learning between the two LoRA factors and justify whether one initialization is preferable to another. Left: The norm of the two factors over training iterations. Middle: The Pearson correlation over training iterations. Note that even though the norm of $\bm A$ remains nearly constant throughout training, the test performance, as measured by the Pearson correlation, still indicates good accuracy. Right: By using discrepant learning rates as in LoRA+ hayou2024lora, we obtain faster convergence.

Theorems & Definitions (9)

• Example 2.1: Multilayer Perceptron.
• Theorem 1
• Theorem 2
• Corollary 1
• Theorem 3
• Example 4.1: Dropout
• Example 4.2: Masked self-supervised learning
• Theorem 4: GD-GaLore and GD-ReLoRA are equivalent with full rank gradient initialization
• proof