Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing

TL;DR

The paper tackles the problem of achieving high-performance instruction execution without resorting to constantly scaling up LLM size. It introduces capability instruction tuning and the Model-SAT router, which uses a capability encoder and a lightweight LLM to predict, for each candidate model, its aptitude on a given instruction based on a $50$-task, $20$-shot aptitude test. By routing at the instruction level and avoiding per-candidate inferences during deployment, Model-SAT delivers state-of-the-art routing across multiple model zoos, including multimodal extensions, and generalizes to unseen models with only minimal new-task inferences. The work provides extensive benchmarks and an open-source toolkit to enable dynamic, scalable routing in real-world settings, significantly reducing resource use while maintaining or exceeding the performance of much larger models.

Abstract

Large Language Models (LLMs) have demonstrated human-like instruction-following abilities, particularly those exceeding 100 billion parameters. The combined capability of some smaller, resource-friendly LLMs can address most of the instructions that larger LLMs excel at. In this work, we explore how to route the best-performing LLM for each instruction to achieve better overall performance. We develop a new paradigm, constructing capability instructions with model capability representation, user instruction, and performance inquiry prompts to assess the performance. To learn from capability instructions, we introduce a new end-to-end framework called Model Selection with Aptitude Test (Model-SAT), which generates positive and negative samples based on what different models perform well or struggle with. Model-SAT uses a model capability encoder that extends its model representation to a lightweight LLM. Our experiments show that Model-SAT understands the performance dimensions of candidate models and provides the probabilities of their capability to handle various instructions. Additionally, during deployment, a new model can quickly infer its aptitude test results across 50 tasks, each with 20 shots. Model-SAT performs state-of-the-art model routing without candidate inference and in real-world new model-released scenarios. The code is available at https://github.com/Now-Join-Us/CIT-LLM-Routing

Figures (5)

• Figure 1: Illustration of Coverage Observation: The combined capabilities of the earlier-released model zoo effectively address most of the instructions that GPT-4o excels at. The union of samples managed accurately by Phi-1, ChatGLM2, and Zephyr covers 80% of GPT-4o's correct instructions. The smaller-scale model zoo can enhance overall performance by selecting a suitable model for each instruction.
• Figure 2: Illustration of Model Routing with Capability Instructions: A Comparison with Re-ranking Based Methods. The goal of model router is to select the optimal model for a given user instruction without access to ground truth and enhance overall performance. Previous re-ranking methods require inference for each candidate. Model-SAT employs a lightweight aptitude test to create capability representations. It learns the intrinsic relationship between model representations and the instructions to be assigned, significantly speeding up model routing and streamlining deployment.
• Figure 3: One example of a Capability Instruction. It is an instruction for model routing that inquires whether a model can handle a specific user instruction. It comprises three components: the capability representation $\texttt{c}^m$ based on the streamlined aptitude test, the user instruction $\mathbf{x}_i$ to be assigned, and a performance inquiry prompt $\texttt{p}$. This instruction is inputted into the Model-SAT Capability LLM, which outputs the probability that the model can perform the user instruction well.
• Figure 4: The Architecture of Model-SAT.
• Figure 5: Real-world Model Routing with Latest LLM Release on ARC-Challenge. Model-SAT (in red) quickly generalizes to unseen LLMs without extra training, maintaining robust performance despite dynamically adding diverse LLMs.