Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning

TL;DR

SETA tackles the plasticity–stability conflict in continual learning for large language models by decomposing the model into a mix of sparse experts: shared components that capture common representations and unique components that memorize task-specific information. A Split-on-Share mechanism evolves these experts with elastic weight anchoring, while an adaptive, task-agnostic gating network retrieves relevant experts without task identifiers. Gradient-based sparse subspace selection guides where updates occur, and SoS thresholds ensure robust separation of shared and unique knowledge, enabling logarithmic capacity growth and efficient parameter usage. Empirically, SETA outperforms PEFT baselines across diverse domain-specific and general benchmarks, demonstrating strong knowledge retention and improved generalization with minimal parameter overhead.

Abstract

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning, referred to as SETA, a framework that resolves the plasticity-stability conflict by decomposing the model into modular subspaces. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through elastic weight anchoring, which protects critical shared knowledge and enables a unified gating network to automatically retrieve the correct expert combination for each task during inference. Extensive experiments across diverse domain-specific and general benchmarks demonstrate that SETA consistently outperforms state-of-the-art parameter-efficient fine-tuning-based continual learning methods.

Figures (9)

• Figure 1: Overview of the SETA Framework Architecture.(a--b) Sparse Subspace Selection: High-utility parameter blocks are identified from the pre-trained LLM using gradient magnitude to form the expert design. (c) Split-on-Share (SoS) Evolution: The SoS filter partitions parameters into plastic Shared ($E_{\text{s}}$) and frozen Unique ($E_{\text{u}}$) experts to resolve parameter collisions and retain knowledge. (d) Gating Evaluation: The gating network expands using logit invariance to strictly preserve decision boundaries during expert splitting. (e) Task-Agnostic Inference: A router network dynamically selects the top-$k$ experts for input tokens, enabling automatic task processing without task identifications.
• Figure 2: Average absolute gradient magnitudes. The $4096 \times 4096$ weight matrices are partitioned into a $16 \times 16$ grid ($l=256$), revealing structured sparsity.
• Figure 3: Layer-wise analysis shows $\sim$95% of top-ranked gradients concentrate in the Value (V) projection, indicating that attention routing (Q, K) remains stable while content (V) requires adaptation.
• Figure 4: Aggregated selection frequency of the V-projection blocks. Distinct distributional shifts suggest potential for interference despite shared high activity.
• Figure 5: Subspace Evolution. The shared expert (blue) progressively absorbs universal features, while task-specific unique experts (colored bands) maintain stable, minimal footprints.