MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs

TL;DR

MetaLLM addresses the problem of selecting among multiple LLMs to optimize accuracy under a cost constraint for zero-shot classification and MCQA. It defines a per-query reward $r(x,i) = a_i(x) - p c_i$ and learns a linear reward model $Q_j(x; theta_j)$ to estimate expected rewards, updating via ridge regression. It uses an Upper Confidence Bound strategy to select the LLM and updates rewards online, with input features derived from a Sentence-BERT embedding. Experiments on OpenAI and Together AI show MetaLLM achieves about a 1% accuracy gain and substantial cost reductions (up to around 60% on OpenAI and 10% on Together AI) over single-model baselines, illustrating a practical, lightweight approach to cost-aware deployment and extending to other tasks via reward function design.

Abstract

The rapid progress in machine learning (ML) has brought forth many large language models (LLMs) that excel in various tasks and areas. These LLMs come with different abilities and costs in terms of computation or pricing. Since the demand for each query can vary, e.g., because of the queried domain or its complexity, defaulting to one LLM in an application is not usually the best choice, whether it is the biggest, priciest, or even the one with the best average test performance. Consequently, picking the right LLM that is both accurate and cost-effective for an application is necessary yet remains a challenge. In this paper, we introduce MetaLLM, a framework that dynamically and intelligently routes each query to the optimal LLM (among several available LLMs) for classification and multi-choice question-answering tasks, achieving significantly improved accuracy and cost-effectiveness. By framing the selection problem as a multi-armed bandit, MetaLLM balances prediction accuracy and cost efficiency under uncertainty. Our experiments, conducted on popular LLM platforms such as OpenAI and Together AI, as well as open-source LLM, showcase MetaLLM's efficacy in real-world scenarios, laying the groundwork for future extensions.

Figures (6)

• Figure 1: The general process of serving queries in MetaLLM. MetaLLM wraps around an existing LLM Provider, inspects each query, and then routes it to the least expensive LLM that can provide an accurate response. As an example, the blue query can be answered accurately by LLM 1 and LLM 2, but MetaLLM will route it to LLM 1 since it is less expensive; similarly in another example, the green query is routed to the least expensive LLM 2 even though LLMs 2 and 3 both can accurately answer it. The entire process is lightweight and can be performed without accessing in LLMs.
• Figure 2: The number of samples that can be answered by one model but not by the other models. Cheaper models can answer many queries that more expensive models cannot.
• Figure 3: The cost-accuracy trade-off of MetaLLM with the cost in log scale for better visualization. MetaLLM with dynamic cost in the reward function slightly decreases the accuracy with a high budget; however, both approaches can perform better than a single LLM.
• Figure 4: The histogram of OpenAI LLMs selected by MetaLLM with $p=0$.
• Figure 5: The histogram of OpenAI LLMs selected by MetaLLM with $p=0.001$.