OptLLM: Optimal Assignment of Queries to Large Language Models

TL;DR

OptLLM addresses the challenge of deploying multiple LLMs under cost constraints by formulating a two-objective allocation problem that minimizes total cost while maximizing query success. It combines a robust bootstrap-based prediction module to estimate per-query LLM performance with an optimization component that initializes two extreme solutions and iteratively destructs and reconstructs to generate a Pareto front of high-quality allocations. The approach demonstrates cost reductions of 2.40% to 49.18% without sacrificing accuracy and outperforms classic MO algorithms in both front quality (IGD) and diversity (Δ), while also reducing runtime. The framework is validated across diverse NLP tasks and a domain-specific log-parsing benchmark, shows ablations confirming the importance of optimization and robust prediction, and provides practical, publicly accessible results and code for real-world deployment.

Abstract

Large Language Models (LLMs) have garnered considerable attention owing to their remarkable capabilities, leading to an increasing number of companies offering LLMs as services. Different LLMs achieve different performance at different costs. A challenge for users lies in choosing the LLMs that best fit their needs, balancing cost and performance. In this paper, we propose a framework for addressing the cost-effective query allocation problem for LLMs. Given a set of input queries and candidate LLMs, our framework, named OptLLM, provides users with a range of optimal solutions to choose from, aligning with their budget constraints and performance preferences, including options for maximizing accuracy and minimizing cost. OptLLM predicts the performance of candidate LLMs on each query using a multi-label classification model with uncertainty estimation and then iteratively generates a set of non-dominated solutions by destructing and reconstructing the current solution. To evaluate the effectiveness of OptLLM, we conduct extensive experiments on various types of tasks, including text classification, question answering, sentiment analysis, reasoning, and log parsing. Our experimental results demonstrate that OptLLM substantially reduces costs by 2.40% to 49.18% while achieving the same accuracy as the best LLM. Compared to other multi-objective optimization algorithms, OptLLM improves accuracy by 2.94% to 69.05% at the same cost or saves costs by 8.79% and 95.87% while maintaining the highest attainable accuracy.

Figures (5)

• Figure 1: Comparison of using each individual LLM with solutions by OptLLM. OptLLM can provide solutions with higher accuracy and lower cost than the best individual LLM (Mixtral 8x7B) on LLM-based log parsing task
• Figure 2: Examples of log parsing and text classification queries
• Figure 3: The Framework of OptLLM
• Figure 4: Solutions by destruction and reconstruction in optimization component. $B$ is a feasible solution derived from the non-dominated solution $A$ by releasing sufficient cost, and $C$ is a new non-dominated solution obtained by reconstructing $B$.
• Figure 5: Ablation study on AGNEWS. The optimization component and robust-aware prediction help find the solution with higher accuracy. The solution with the highest accuracy by OptLLM$_{\text{w/o o}}$ is 69.54%, OptLLM$_{\text{w/o r}}$ is 83.67%, OptLLM is 88.74%

Theorems & Definitions (4)

• Lemma 4.1
• proof
• Lemma 4.2
• proof