Effective LoRA Adapter Routing using Task Representations

TL;DR

LoRAuter tackles the challenge of routing among large, public LoRA adapter pools by moving routing to the level of semantic tasks rather than individual adapters. It builds a task database from lightweight validation sets, assigns best adapters per task via tournament-based evaluation, and uses task embeddings to retrieve the most relevant tasks for a given query, followed by input-aware fusion of the corresponding adapters in the output space. The framework is training-free and black-box, scalable with the number of tasks, and robust to thousands of adapters, achieving near-oracle performance in-domain and strong gains out-of-domain. Empirically it demonstrates strong results across model sizes and adapter pools, including 1.5k+ adapters, while offering efficiency gains through Successive Halving for adapter selection and by operating at the task level rather than per-adapter routing.

Abstract

Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.

Figures (10)

• Figure 1: The LoRAuter framework.
• Figure 2: Average normalized performance of different routing methods on non-OOD and OOD tasks for both models.
• Figure 3: Performance in the semi-OOD setting where validation data is available but the task-specific adapter is removed.
• Figure 4: Normalized performance of LoRAuter with varying number of clusters $K$.
• Figure 5: Normalized performance of uniform selection vs. Successive Halving (SH) across evaluation budgets for Llama2-7B.