Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feedback Alignment Meets Low-Rank Manifolds: A Structured Recipe for Local Learning

Arani Roy, Marco P. Apolinario, Shristi Das Biswas, Kaushik Roy

TL;DR

This work tackles the memory and computation bottlenecks of backpropagation and the scalability limits of Direct Feedback Alignment by introducing SVD-Space Alignment (SSA), a structured local learning framework that trains networks in low-rank weight manifolds defined by Singular Value Decomposition. SSA updates the SVD components ($U$, $S$, $V^T$) with a composite local loss that enforces task performance ($L_{CE}$), alignment with structured feedback ($L_{align}$), and orthogonality preservation ($L_{ortho}$), while maintaining forward and feedback pathways on correspondingly shaped low-rank representations. A dynamic rank reduction strategy compresses the model during training, yielding compact inference without post hoc pruning. Theoretical analysis shows positive gradient alignment with BP and stable updates on the Stiefel manifold, and empirical results on CIFAR-10/100 and ImageNet demonstrate BP-like accuracy, faster convergence than DFA, and reduced memory/compute, especially in deeper and convolutional architectures. Overall, SSA provides a principled, scalable alternative to full-rank gradient-based training for resource-constrained settings, expanding the practicality of local learning in deep networks.

Abstract

Training deep neural networks (DNNs) with backpropagation (BP) achieves state-of-the-art accuracy but requires global error propagation and full parameterization, leading to substantial memory and computational overhead. Direct Feedback Alignment (DFA) enables local, parallelizable updates with lower memory requirements but is limited by unstructured feedback and poor scalability in deeper architectures, specially convolutional neural networks. To address these limitations, we propose a structured local learning framework that operates directly on low-rank manifolds defined by the Singular Value Decomposition (SVD) of weight matrices. Each layer is trained in its decomposed form, with updates applied to the SVD components using a composite loss that integrates cross-entropy, subspace alignment, and orthogonality regularization. Feedback matrices are constructed to match the SVD structure, ensuring consistent alignment between forward and feedback pathways. Our method reduces the number of trainable parameters relative to the original DFA model, without relying on pruning or post hoc compression. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that our method achieves accuracy comparable to that of BP. Ablation studies confirm the importance of each loss term in the low-rank setting. These results establish local learning on low-rank manifolds as a principled and scalable alternative to full-rank gradient-based training.

Feedback Alignment Meets Low-Rank Manifolds: A Structured Recipe for Local Learning

TL;DR

This work tackles the memory and computation bottlenecks of backpropagation and the scalability limits of Direct Feedback Alignment by introducing SVD-Space Alignment (SSA), a structured local learning framework that trains networks in low-rank weight manifolds defined by Singular Value Decomposition. SSA updates the SVD components (, , ) with a composite local loss that enforces task performance (), alignment with structured feedback (), and orthogonality preservation (), while maintaining forward and feedback pathways on correspondingly shaped low-rank representations. A dynamic rank reduction strategy compresses the model during training, yielding compact inference without post hoc pruning. Theoretical analysis shows positive gradient alignment with BP and stable updates on the Stiefel manifold, and empirical results on CIFAR-10/100 and ImageNet demonstrate BP-like accuracy, faster convergence than DFA, and reduced memory/compute, especially in deeper and convolutional architectures. Overall, SSA provides a principled, scalable alternative to full-rank gradient-based training for resource-constrained settings, expanding the practicality of local learning in deep networks.

Abstract

Training deep neural networks (DNNs) with backpropagation (BP) achieves state-of-the-art accuracy but requires global error propagation and full parameterization, leading to substantial memory and computational overhead. Direct Feedback Alignment (DFA) enables local, parallelizable updates with lower memory requirements but is limited by unstructured feedback and poor scalability in deeper architectures, specially convolutional neural networks. To address these limitations, we propose a structured local learning framework that operates directly on low-rank manifolds defined by the Singular Value Decomposition (SVD) of weight matrices. Each layer is trained in its decomposed form, with updates applied to the SVD components using a composite loss that integrates cross-entropy, subspace alignment, and orthogonality regularization. Feedback matrices are constructed to match the SVD structure, ensuring consistent alignment between forward and feedback pathways. Our method reduces the number of trainable parameters relative to the original DFA model, without relying on pruning or post hoc compression. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that our method achieves accuracy comparable to that of BP. Ablation studies confirm the importance of each loss term in the low-rank setting. These results establish local learning on low-rank manifolds as a principled and scalable alternative to full-rank gradient-based training.

Paper Structure

This paper contains 21 sections, 19 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Works
  4. Methodology
  5. Embedding Structural Priors: Breaking Weights into SVD-Space
  6. Local Loss Objective Design
  7. SSA Updates in SVD-Space
  8. Dynamic Rank Reduction Strategy
  9. Theoretical Foundations and Alignment Properties in SSA
  10. SSA Local Loss Properties
  11. Orthogonality-Preserving Updates
  12. Alignment Between SSA and BP Gradients
  13. Gradient Alignment Analysis
  14. Layerwise Alignment Analysis over Time
  15. Experimental Setup
  16. ...and 6 more sections

Figures (5)

  • Figure 1: A comparison of neural network training methods. Notations: $W$ = forward weights, $Layer$ = Layer Activations, $L_{CE}$ = cross-entropy loss, $B$ = random feedback weights, $\delta$ = gradients, $\delta_l$ = gradient of $L_{CE}$ = feedback error, $U,S,V^T$ = SVD components of forward weights, $B_U, B_S, B_{V^T}$ = SVD components of feedback weights, Local Loss (LL) has components: $L_{align}$ = Alignment loss, $L_{ortho}$ = Singular Vector Orthogonality regularizer. (a) Backpropagation (BP): Global gradient updates through each layer. (b) Direct Feedback Alignment (DFA): Local updates with random feedback (c) SVD-space Alignment (SSA) (ours): Decomposes weights into SVD components before training and aligns feedback on those decomposed components itself with local losses. Note: LL + $L_{CE}$ is the total local loss objective.
  • Figure 2: Gradient alignment (angle in degrees) between SSA/DFA and BP gradients across epochs on a 3-layer MLP. Lower angles indicate better alignment.
  • Figure 3: Matrix alignment angles across ResNet-32 layers over training epochs, between (a) SSA (b) DFA, and BP
  • Figure 4: Top-1 Error (%) Across Epochs for a 3-layer MLP.
  • Figure 5: Compute and memory cost comparison of BP, DFA, and SSA across different architectures. SSA achieves lowest inference-time cost through structured low-rank training.