Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalization

Ziyi Han, Huanyu Wang, Zeyu Zhang, Xiangxiang Dai, Xutong Liu, John C. S. Lui

TL;DR

HiLoRA tackles training-free domain generalization for large language models by routing over a pool of LoRAs at two granularities. It models each LoRA as a Gaussian distribution over input embeddings and uses sequence-level Gaussian likelihoods to select LoRAs and allocate a ROC budget, then refines at the token level by locking to informative ROCs via their down-projection vectors. Theoretical error bounds for LoRA identification in ID and OOD regimes guarantee high-probability retention of relevant LoRAs and robust generalization. Empirically, HiLoRA achieves substantial improvements over state-of-the-art baselines across multiple models and domains, with significant accuracy gains and maintained throughput, demonstrating a scalable, training-free path to leveraging community LoRAs for cross-domain robustness.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a widely used technique for adapting large language models (LLMs) to new domains, due to its modular design and broad availability on platforms such as HuggingFace. This availability has motivated efforts to reuse existing LoRAs for domain generalization. However, existing methods often rely on explicit task labels or additional training, which are impractical for deployment. Moreover, they typically activate a fixed number of entire LoRA modules, leading to parameter redundancy or insufficiency that degrade performance. In this paper, we propose \texttt{HiLoRA}, a training-free framework that performs adaptive hierarchical routing over LoRA pools. Drawing on structural properties of LoRA, we define rank-one components (ROCs), in which each rank parameter is regarded as an independent unit. For a given input sequence, \texttt{HiLoRA} first adaptively selects a subset of LoRAs and determines their ROC allocation based on Gaussian likelihoods at the sequence level. At the token level, it further refines routing by activating only the most informative ROCs. We further provide theoretical guarantees that \texttt{HiLoRA} selects the most relevant LoRAs with high probability. Extensive experiments show that \texttt{HiLoRA} achieves substantial improvements in domain generalization, with accuracy gains of up to {\small $55\%$} over state-of-the-art baselines, while maintaining comparable inference throughput.

HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalization

TL;DR

HiLoRA tackles training-free domain generalization for large language models by routing over a pool of LoRAs at two granularities. It models each LoRA as a Gaussian distribution over input embeddings and uses sequence-level Gaussian likelihoods to select LoRAs and allocate a ROC budget, then refines at the token level by locking to informative ROCs via their down-projection vectors. Theoretical error bounds for LoRA identification in ID and OOD regimes guarantee high-probability retention of relevant LoRAs and robust generalization. Empirically, HiLoRA achieves substantial improvements over state-of-the-art baselines across multiple models and domains, with significant accuracy gains and maintained throughput, demonstrating a scalable, training-free path to leveraging community LoRAs for cross-domain robustness.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a widely used technique for adapting large language models (LLMs) to new domains, due to its modular design and broad availability on platforms such as HuggingFace. This availability has motivated efforts to reuse existing LoRAs for domain generalization. However, existing methods often rely on explicit task labels or additional training, which are impractical for deployment. Moreover, they typically activate a fixed number of entire LoRA modules, leading to parameter redundancy or insufficiency that degrade performance. In this paper, we propose \texttt{HiLoRA}, a training-free framework that performs adaptive hierarchical routing over LoRA pools. Drawing on structural properties of LoRA, we define rank-one components (ROCs), in which each rank parameter is regarded as an independent unit. For a given input sequence, \texttt{HiLoRA} first adaptively selects a subset of LoRAs and determines their ROC allocation based on Gaussian likelihoods at the sequence level. At the token level, it further refines routing by activating only the most informative ROCs. We further provide theoretical guarantees that \texttt{HiLoRA} selects the most relevant LoRAs with high probability. Extensive experiments show that \texttt{HiLoRA} achieves substantial improvements in domain generalization, with accuracy gains of up to {\small } over state-of-the-art baselines, while maintaining comparable inference throughput.
Paper Structure (23 sections, 5 theorems, 22 equations, 8 figures, 5 tables)

This paper contains 23 sections, 5 theorems, 22 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. HiLoRA Framework
  5. Input-Aware ROC Allocation
  6. Token-level ROC Routing
  7. Theoretical Analysis
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Further Analysis
  12. Conclusion
  13. Related Work
  14. Theoretical Demonstration
  15. ariance Normalization Property
  16. ...and 8 more sections

Key Result

Lemma 1

For any two distributions $i,j$ with class-conditional Gaussians $\mathcal{N}(\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i)$ and $\mathcal{N}(\boldsymbol{\mu}_j,\boldsymbol{\Sigma}_j)$ and prior probabilities $\pi_i,\pi_j$, the Bayes error rate satisfies: $P_{\rm err}^{(2)}(i,j) \;\le\; \sqrt{\pi_i\pi_j

Figures (8)

  • Figure 1: Average accuracy over ten NLI tasks, with five seen tasks and five unseen tasks. HiLoRA achieves the best performance and approaches the accuracy of task-specific LoRAs. Detailed results are shown in Tab. \ref{['tab:within_cluster']}.
  • Figure 2: Scatter plots of the first two principal components derived from vectors in LoRA projection matrices specialized for five NLI tasks. The boxes highlight examples where optimal routing for an unseen task (pink) would involve selecting only the vectors aligned with relevant semantics.
  • Figure 3: Overview of HiLoRA architecture.
  • Figure 4: Input-LoRA similarity heatmap produced by HiLoRA, where tasks from the same cluster are enclosed within green boxes for clarity.
  • Figure 5:
  • ...and 3 more figures

Theorems & Definitions (5)

  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Lemma 2: Theorem 3.1 in zhao2025merging
  • Lemma 3