Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models

Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, Jingren Zhou

TL;DR

The paper tackles the high computational cost of ensembling large language models via reward-model ranking by introducing Zooter, a reward-guided routing method that distills rewards from off-the-shelf reward models to train a lightweight routing function. Zooter uses tag-based label enhancement to reduce reward noise and routes each query to a single expert LLM, achieving competitive or superior performance across 26 benchmark subsets with significantly lower inference cost than traditional reward-model ranking. The approach reveals latent, domain-specific expertise across diverse LLMs and offers a data-efficient, scalable path to practical ensemble deployment. Future work could deepen interpretability of the learned routing and the nature of cross-model expertise.

Abstract

The complementary potential of Large Language Models (LLM) assumes off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and tasks so that an ensemble of LLMs can achieve consistently better performance. Existing ensemble methods for LLMs mainly focus on reward model ranking of outputs, leading to significant computation overhead. To combat this issue, we revisit the complementary potential of LLMs and further elaborate it by mining latent expertise with off-the-shelf reward models. We propose Zooter, a reward-guided routing method distilling rewards on training queries to train a routing function, which can precisely distribute each query to the LLM with expertise about it. We also integrate a tag-based label enhancement to mitigate noise from uncertainty when using rewards as silver supervision. Zooter shows computation efficiency in inference as it introduces only a minor computation overhead of a routing function compared with reward model ranking methods. We evaluate Zooter on a comprehensive benchmark collection with 26 subsets on different domains and tasks. Zooter outperforms the best single model on average and ranks first on 44% of tasks, even surpassing multiple reward model ranking methods.

Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models

TL;DR

The paper tackles the high computational cost of ensembling large language models via reward-model ranking by introducing Zooter, a reward-guided routing method that distills rewards from off-the-shelf reward models to train a lightweight routing function. Zooter uses tag-based label enhancement to reduce reward noise and routes each query to a single expert LLM, achieving competitive or superior performance across 26 benchmark subsets with significantly lower inference cost than traditional reward-model ranking. The approach reveals latent, domain-specific expertise across diverse LLMs and offers a data-efficient, scalable path to practical ensemble deployment. Future work could deepen interpretability of the learned routing and the nature of cross-model expertise.

Abstract

The complementary potential of Large Language Models (LLM) assumes off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and tasks so that an ensemble of LLMs can achieve consistently better performance. Existing ensemble methods for LLMs mainly focus on reward model ranking of outputs, leading to significant computation overhead. To combat this issue, we revisit the complementary potential of LLMs and further elaborate it by mining latent expertise with off-the-shelf reward models. We propose Zooter, a reward-guided routing method distilling rewards on training queries to train a routing function, which can precisely distribute each query to the LLM with expertise about it. We also integrate a tag-based label enhancement to mitigate noise from uncertainty when using rewards as silver supervision. Zooter shows computation efficiency in inference as it introduces only a minor computation overhead of a routing function compared with reward model ranking methods. We evaluate Zooter on a comprehensive benchmark collection with 26 subsets on different domains and tasks. Zooter outperforms the best single model on average and ranks first on 44% of tasks, even surpassing multiple reward model ranking methods.
Paper Structure (11 sections, 5 equations, 3 figures, 3 tables)

This paper contains 11 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Complementary Potential of LLMs
  5. Zooter
  6. Experiments
  7. Experimental Setup
  8. Results
  9. Analysis
  10. Conclusion
  11. Datasets

Figures (3)

  • Figure 1: An example of the large language model ensemble. Reward model ranking marked in blue needs to generate responses from all models while Zooter routers the given query to the best model and only infers one model. This case is collected from the MT-Bench benchmark and we also present oracle judgements of each response.
  • Figure 2: Overview of Zooter. Zooter aims to assemble a set of off-the-shelf LLMs by first conducting a reward model ranking on a diverse training set to obtain supervision of model expertise, highlighted in blue in the figure. Instruction tags are then used to mitigate the uncertainty in reward estimation. Zooter uses the normalized rewards as supervision to train a routing function by knowledge distillation. The training circle is marked in green, and the inference is marked in orange. Zooter is much lighter in computation as it routes the query to the corresponding expert LLM during inference time, while reward model ranking has to generate outputs for all candidates.
  • Figure 3: Analysis between reward entropy and scores of reward preference ranking on MT-bench.