SAL: Selective Adaptive Learning for Backpropagation-Free Training with Sparsification

TL;DR

This work addresses the biological implausibility and gradient-interference limitations of Backpropagation by introducing Selective Adaptive Learning (SAL), a BP-free training paradigm that combines selective parameter activation with adaptive area partitioning to decouple the parameter space into mutually exclusive, sample-dependent regions. SAL uses a Learned-Frozen Decoupled Routing mechanism (learnable feature projection, fixed prototype anchors, and hard area routing) to route each input to a single area and perform area-conditioned forward passes with asymmetric, fixed feedback for local learning signals. The method achieves competitive convergence on ten benchmarks, scales to deep regimes (up to 128 layers) and large models (up to 1B parameters), and demonstrates stability and sparsity advantages compared to BP and MoE baselines. SAL thus provides a biologically inspired, scalable alternative for training deep networks without end-to-end gradient propagation, with potential implications for energy-efficient and hardware-friendly learning. The work also highlights future directions for extending area-based routing to other architectures and refining routing efficiency and memory usage.

Abstract

Standard deep learning relies on Backpropagation (BP), which is constrained by biologically implausible weight symmetry and suffers from significant gradient interference within dense representations. To mitigate these bottlenecks, we propose Selective Adaptive Learning (SAL), a training method that combines selective parameter activation with adaptive area partitioning. Specifically, SAL decomposes the parameter space into mutually exclusive, sample-dependent regions. This decoupling mitigates gradient interference across divergent semantic patterns and addresses explicit weight symmetry requirements through our refined feedback alignment. Empirically, SAL demonstrates competitive convergence rates, leading to improved classification performance across 10 standard benchmarks. Additionally, SAL achieves numerical consistency and competitive accuracy even in deep regimes (up to 128 layers) and large-scale models (up to 1B parameters). Our approach is loosely inspired by biological learning mechanisms, offering a plausible alternative that contributes to the study of scalable neural network training.

Figures (5)

• Figure 1: A multi-layer network architecture based on SAL.
• Figure 2: Performance comparison between SAL and the baseline across various network depths on 4 datasets. Across all plots, a consistent encoding is used: colors distinguish datasets, and line styles differentiate methods (e.g., solid for SAL, dotted for baseline).
• Figure 3: Performance comparison between SAL and the baseline across various network widths on 4 datasets.
• Figure 4: Accuracy comparison between SAL and MoE across various networks and 4 datasets. The MoE architecture follows the same structure with Table \ref{['tab:main_final_test_accuracy_sal_vs_baseline']}, where the $n_{\text{areas}}$ parameter from SAL serves as the number of experts.
• Figure 5: Acucracy comparison between SAL and MoE across various networks and 4 datasets.