SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees

TL;DR

SMART, Scaling Models Adaptively for Reduced Token Fees, a novel LLM framework designed to minimize the inference costs of NLP tasks while ensuring sufficient result quality, is introduced.

Abstract

The advancement of Large Language Models (LLMs) has significantly boosted performance in natural language processing (NLP) tasks. However, the deployment of high-performance LLMs incurs substantial costs, primarily due to the increased number of parameters aimed at enhancing model performance. This has made the use of state-of-the-art LLMs more expensive for end-users. AI service providers, such as OpenAI and Anthropic, often offer multiple versions of LLMs with varying prices and performance. However, end-users still face challenges in choosing the appropriate LLM for their tasks that balance result quality with cost. We introduce SMART, Scaling Models Adaptively for Reduced Token Fees, a novel LLM framework designed to minimize the inference costs of NLP tasks while ensuring sufficient result quality. It enables users to specify an accuracy constraint in terms of the equivalence of outputs to those of the most powerful LLM. SMART then generates results that deviate from the outputs of this LLM only with a probability below a user-defined threshold. SMART employs a profiling phase that evaluates the performance of multiple LLMs to identify those that meet the user-defined accuracy level. SMART optimizes the tradeoff between profiling overheads and the anticipated cost savings resulting from profiling. Moreover, our approach significantly reduces inference costs by strategically leveraging a mix of LLMs. Our experiments on three real-world datasets show that, based on OpenAI models, SMART achieves significant cost savings, up to 25.6x in comparison to GPT-4.

Figures (10)

• Figure 1: Costs of OpenAI LLMs for the sentiment classification task on the IMDB benchmark.
• Figure 2: Overview of Smart.
• Figure 3: Costs of GPT-4 and all variants of Smart on all benchmarks.
• Figure 4: Costs of GPT-4 and all variants of Smart with varying accuracy constraint $\delta$.
• Figure 5: Breakdown of Smart in terms of the number of instances processed by each LLM. The total number of instances processed by LLMs may exceed the number of instances in the dataset due to profiling, where the same item is processed using multiple LLMs.

Theorems & Definitions (3)

• Definition 1: LLMs with Varying Performance and Costs
• Definition 2: $\langle \delta, \gamma \rangle$-Equivalence
• Definition 3: Task-Specific LLM Inference with $\langle \delta, \gamma \rangle$-Equivalence Guarantees