Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoLoRA: Composable Specialization via Per-Token Adapter Routing

Shrey Shah, Justin Wagle

Abstract

Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like "write code to solve this equation," which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.

MoLoRA: Composable Specialization via Per-Token Adapter Routing

Abstract

Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like "write code to solve this equation," which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.
Paper Structure (86 sections, 4 theorems, 11 equations, 9 figures, 17 tables, 2 algorithms)

This paper contains 86 sections, 4 theorems, 11 equations, 9 figures, 17 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem 1: Multimodal Efficiency.
  3. Problem 2: Mixed-Capability Quality.
  4. Contributions.
  5. Background and Related Work
  6. Low-Rank Adaptation
  7. Multi-Adapter Serving
  8. Mixture of Experts
  9. Multimodal LLMs and Vocabulary Structure
  10. Per-Token Routing Framework
  11. From Sequences to Tokens
  12. Assumption.
  13. Compositional Routing
  14. Unified Dispatch Infrastructure
  15. Theoretical Analysis
  16. ...and 71 more sections

Key Result

Theorem 3.4

Vocabulary routing achieves $\mathcal{O}(1)$ per-token routing cost for fixed $M$, compared to $\mathcal{O}(E \cdot d)$ for learned MoE gating over $E$ experts with hidden dimension $d$.

Figures (9)

  • Figure 1: Per-sequence routing (left) sends all tokens in a sequence to the same adapter, even when tokens have different modalities (T=text, I=image, A=audio, V=video). Per-token routing (right) routes each token to its modality-specialized adapter within a single forward pass.
  • Figure 2: MoLoRA: logical vs. physical execution.Left: Per-token view---a router selects top-$k$ adapters (here $k$=2), which are combined via weighted sum. Right: Batched execution---tokens are grouped by their selected adapter, enabling parallel grouped GEMM using identical infrastructure to MoE systems.
  • Figure 3: MoLoRA enables small models to exceed larger ones. Qwen3-1.7B with four specialized LoRA adapters and learned routing (blue) exceeds Qwen3-8B (red) on all four reasoning benchmarks, while being 4.7$\times$ smaller.
  • Figure 4: System architecture comparison. Paging (left) copies adapters on-demand from CPU, causing variable latency and preventing CUDA graph capture. Our hot-set architecture (right) pre-allocates GPU-resident adapters with fixed addresses, enabling graph capture and predictable latency.
  • Figure 5: Kernel performance comparison. Scalar implementations (S-LoRA's BGMV) excel in the memory-bound regime (batch $<$160), while our tensor-core kernel with CUDA graph capture dominates in the compute-bound regime. The crossover occurs at batch size $\approx$160, with our kernel achieving 1.98$\times$ speedup at batch 512.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Definition 3.1: Vocabulary Routing
  • Definition 3.2: Compositional Routing
  • Example 3.3
  • Theorem 3.4: Routing Complexity
  • Theorem 3.5: Expressiveness
  • Theorem 3.6: Computational Optimality
  • Theorem 3.7: Sparse Attention Equivalence
  • proof : Proof sketch
  • Definition 5.1: Hot-Set Layout