Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Routoo: Learning to Route to Large Language Models Effectively

Alireza Mohammadshahi, Arshad Rafiq Shaikh, Majid Yazdani

TL;DR

The paper tackles the high cost of deploying large language models by proposing Routoo, a routing architecture that leverages a pool of existing LLMs to balance quality and inference cost. It introduces a performance predictor to estimate each model's suitability for a given prompt and a cost-aware selector to assign queries to models under a budget, augmented by a universe constructor to build a complementary model set. Through experiments on the MMLU benchmark, Routoo matches Mixtral 8x7b at lower cost and, with GPT-4 integrated, approaches GPT-4 performance at reduced expense, sometimes surpassing it at similar budgets. The approach suggests a scalable, cost-efficient path to high-quality language reasoning by orchestrating multiple LLMs rather than training new foundations from scratch.

Abstract

LLMs with superior response quality--particularly larger or closed-source models--often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement. Routoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. We evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5% at equivalent costs, achieving an accuracy of 75.9%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25% cost reduction. These outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs.

Routoo: Learning to Route to Large Language Models Effectively

TL;DR

The paper tackles the high cost of deploying large language models by proposing Routoo, a routing architecture that leverages a pool of existing LLMs to balance quality and inference cost. It introduces a performance predictor to estimate each model's suitability for a given prompt and a cost-aware selector to assign queries to models under a budget, augmented by a universe constructor to build a complementary model set. Through experiments on the MMLU benchmark, Routoo matches Mixtral 8x7b at lower cost and, with GPT-4 integrated, approaches GPT-4 performance at reduced expense, sometimes surpassing it at similar budgets. The approach suggests a scalable, cost-efficient path to high-quality language reasoning by orchestrating multiple LLMs rather than training new foundations from scratch.

Abstract

LLMs with superior response quality--particularly larger or closed-source models--often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement. Routoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. We evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5% at equivalent costs, achieving an accuracy of 75.9%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25% cost reduction. These outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs.
Paper Structure (26 sections, 7 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 26 sections, 7 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mixture-of-Experts.
  4. Model Selection.
  5. Architecture
  6. Problem Formulation
  7. Performance Predictor
  8. Cost-Aware Selector
  9. Universe Constructor
  10. Results and Discussion
  11. Experiment Setting
  12. Evaluation Setting.
  13. Baselines.
  14. Architecture Setting.
  15. Training Data Preparation
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Routoo architecture has three main components: a performance predictor, cost-aware selector, and universe constructor. The universe constructor identifies the optimal set of complementary models from all available models. The performance predictor predicts the correctness of experts for a specified query, and the cost-aware selector chooses the underlying model by considering the cost and efficiency of each model.
  • Figure 2: The performance of different Routoo models and baselines on MMLU benchmark, given different budget limitations.
  • Figure 3: Per sub-category performances of our Routoo models and baselines on MMLU benchmark.
  • Figure 4: Routing distributions of different variants of Routoo models. The inference cost is provided for 1 million tokens.