Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion

Vladimer Khasia

TL;DR

MoE architectures enable scaling but incur memory bandwidth and optimization instability when routing to many adapters. Dynamic Subspace Composition introduces a sparse dictionary-like weight update built from a shared bank of rank-1 atoms, enabling high rank context dependent updates with parameter complexity $O(Md)$ and memory traffic $O(Kd)$. A star shaped residual domain with Magnitude-Gated Simplex routing maintains continuity at the identity and provides contraction when routing confidence is low, complemented by frame potential regularization and spectral bounds for stability. On WikiText-103, DSC matches standard MoE performance while delivering roughly $15\%$ lower inference latency under Iso-Active budgets, demonstrating practical gains for scalable conditional computation.

Abstract

Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC

Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion

TL;DR

MoE architectures enable scaling but incur memory bandwidth and optimization instability when routing to many adapters. Dynamic Subspace Composition introduces a sparse dictionary-like weight update built from a shared bank of rank-1 atoms, enabling high rank context dependent updates with parameter complexity and memory traffic . A star shaped residual domain with Magnitude-Gated Simplex routing maintains continuity at the identity and provides contraction when routing confidence is low, complemented by frame potential regularization and spectral bounds for stability. On WikiText-103, DSC matches standard MoE performance while delivering roughly lower inference latency under Iso-Active budgets, demonstrating practical gains for scalable conditional computation.

Abstract

Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC
Paper Structure (18 sections, 1 theorem, 14 equations, 1 figure, 2 tables, 2 algorithms)

This paper contains 18 sections, 1 theorem, 14 equations, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Notation and Geometric Interpretation
  4. Projected Basis Decomposition
  5. Coordinate Generation: Magnitude-Gated Simplex
  6. Spectral Analysis and Stability
  7. Optimization Objective and Rigorous Regularization
  8. Complexity and Factorization
  9. Algorithm Specification
  10. Experiments
  11. Experimental Setup
  12. Architectural Configurations
  13. Results and Analysis
  14. Conclusion
  15. Experimental Details
  16. ...and 3 more sections

Key Result

Proposition 1

Let $\|\mathbf{u}_j\|_2 \le 1$ and $\|\mathbf{v}_j\|_2 \le 1$. Consequently, the spectral norm of each basis atom is bounded: $\|\mathbf{u}_j^\top \mathbf{v}_j\|_2 \le 1$. The Lipschitz constant of the residual branch is bounded as follows: Case 1: Global Scalar Scaling (Algorithm alg:dmc). Let $\ga Case 2: Channel-Wise Spectral Relaxation (Algorithm alg:dsc_final). Let $\bm{\gamma} \in \mathbb{R

Figures (1)

  • Figure 1: Training Convergence. Validation loss curves over 2,000 iterations. DSC (Red) follows the trajectory of Standard MoE (Blue) closely, rapidly diverging from the Dense baseline (Green). Shaded regions indicate standard deviation across random seeds ($42, 1337$).

Theorems & Definitions (3)

  • Definition 1: $\ell_2$-Projected Normalization
  • Remark 1: Continuity and Contractive Terminology
  • Proposition 1: Conservative Lipschitz Bound