Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning

Abstract

We present Brainstacks, a modular architecture for continual multi-domain fine-tuning of large language models that packages domain expertise as frozen adapter stacks composing additively on a shared frozen base at inference. Five interlocking components: (1) MoE-LoRA with Shazeer-style noisy top-2 routing across all seven transformer projections under QLoRA 4-bit quantization with rsLoRA scaling; (2) an inner loop performing residual boosting by freezing trained stacks and adding new ones; (3) an outer loop training sequential domain-specific stacks with curriculum-ordered dependencies; (4) null-space projection via randomized SVD constraining new stacks to subspaces orthogonal to prior directions, achieving zero forgetting in isolation; (5) an outcome-based sigmoid meta-router trained on empirically discovered domain-combination targets that selectively weights stacks, enabling cross-domain composition. Two boundary experiments: (6) PSN pretraining on a randomly initialized model; (7) per-domain RL (DPO/GRPO) validating compatibility with post-SFT alignment. Validated on TinyLlama-1.1B (4 domains, 9 stacks) and Gemma 3 12B IT (5 domains, 10 stacks), MoE-LoRA achieves 2.5x faster convergence than parameter-matched single LoRA, residual boosting breaks through the single-stack ceiling, and the routed system recovers generation quality destroyed by ungated stack accumulation. The central finding: the outcome-based router discovers that domain stacks encode transferable cognitive primitives (instruction-following clarity, numerical reasoning, procedural logic, chain-of-thought structure) rather than domain-specific knowledge, with medical prompts routing to chat+math stacks in 97% of cases despite zero medical data in those stacks.

Figures (13)

• Figure 1: Domain subspace directions at layer 24 (q_proj). Each arrow is a top-3 principal direction extracted via SVD. Domains point in distinct orientations, confirming the null-space projection forces orthogonal subspace separation. Math (red) and reasoning (orange) share partial overlap (cosine similarity 0.54), consistent with shared training data sources.
• Figure 2: Validation loss curves for Single LoRA (rank 64) vs MoE-LoRA (4 experts, rank 16). MoE-LoRA converges faster per step despite 2x wall-clock overhead.
• Figure 3: Single LoRA vs Brainstacks residual boosting on chat domain. Single LoRA plateaus at 0.8741 after 400 steps. Brainstacks breaks through this ceiling via frozen stacked residuals, reaching 0.8531 after 3 rounds (1200 cumulative steps)---a 2.4% relative improvement. Each round adds a new 53M-parameter stack that learns what the previous frozen stacks left uncaptured.
• Figure 4: Ungated interference matrix (with null-space projection). Each row shows all-domain validation losses after training through that domain. Gray cells indicate domains not yet trained. Chat degrades from 0.853 to 1.778 as stacks accumulate; medical suffers the worst interference at 1.984. These elevated values are not forgetting---frozen weights are unchanged---but magnitude accumulation from ungated cross-stack interference.
• Figure 5: Ungated interference matrix (without null-space projection). Compared to Figure \ref{['fig:fig2']}, all final-stage losses are higher: chat 1.839 (vs 1.778), code 1.512 (vs 1.452), medical 2.066 (vs 1.984), math 1.184 (vs 1.041). Without null-space protection, interference accumulates faster across domains.