HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Search

TL;DR

HAPS addresses the limitation of prior LLM routing methods that only select architectures by introducing a hierarchical routing framework that jointly searches architectures and parameters. A high‑level router selects an LLM architecture, while a low‑level neural generator produces per‑model LoRA parameters, with part of the low‑level network sharing parameters with the high‑level router to enable mutual learning. The two components are trained end‑to‑end via a reward‑augmented objective, enabling discrete (architecture) and continuous (parameter) optimization in a unified framework. Empirical results on HotpotQA and MMLU show HAPS outperforms strong baselines in most settings and remains effective in mixed open/closed‑source deployments, demonstrating practical gains in task adaptation and efficiency.

Abstract

Large language model (LLM) routing aims to exploit the specialized strengths of different LLMs for diverse tasks. However, existing approaches typically focus on selecting LLM architectures while overlooking parameter settings, which are critical for task performance. In this paper, we introduce HAPS, a hierarchical LLM routing framework that jointly searches over model architectures and parameters. Specifically, we use a high-level router to select among candidate LLM architectures, and then search for the optimal parameters for the selected architectures based on a low-level router. We design a parameter generation network to share parameters between the two routers to mutually enhance their capabilities. In the training process, we design a reward-augmented objective to effectively optimize our framework. Experiments on two commonly used benchmarks show that HAPS consistently outperforms strong routing baselines. We have released our code at https://github.com/zihangtian/HAPS.

Figures (14)

• Figure 1: Illustration of the traditional LLM router versus our proposed model. Unlike standard methods that only select architectures, our framework jointly routes both LLM architectures and their specific parameters.
• Figure 2: The overall framework of our hierarchical LLM router. The complete routing process includes two phases: (1) Architecture search based on the high-level router. (2) Parameter search based on the low-level router. The high- and low-level routers share the parameter $\theta$ to enable cross-level knowledge transfer.
• Figure 3: Analysis of the joint optimization strategy. The comparison with the decoupled variant (w/o Parameter Sharing) reveals significant performance degradation in the isolated setting, confirming that parameter sharing is essential for robust, task-aware routing.
• Figure 4: Impact of LoRA depth on performance. We vary the number of adapted layers on the attention output projection (o_proj). Overall, $\mathbf{l}_2$ is the most robust choice: it consistently improves over $\mathbf{l}_1$ while avoiding the degradation at $\mathbf{l}_3$ for L-Q and L-M.
• Figure 5: Performance comparison on mixed-source LLMs (HotpotQA). HAPS consistently outperforms baselines by optimizing the open-source model to synergize with fixed proprietary APIs, demonstrating robust adaptability in hybrid deployments.

Theorems & Definitions (3)

• Remark 1: Extension to mixed open- and closed-source LLM routing
• Remark 2: Loss comparison between traditional and our LLM routers
• Remark 3: Inference Efficiency Analysis