Agreement-Based Cascading for Efficient Inference

TL;DR

This work tackles the high cost of inference with large models by proposing Agreement-Based Cascading (ABC), a training-free, drop-in cascade that routes data to cheaper models based on agreement among ensembles at each tier. ABC leverages simple voting or scoring-based deferral rules to decide when to escalate to larger models, balancing accuracy with inference cost, and it demonstrates competitive or superior accuracy while reducing overall compute. The authors provide formal definitions of safe deferral and prove that ABC can be cost-competitive under realistic conditions, then validate the approach across image and language tasks. Real-world case studies show substantial cost savings in edge-to-cloud, heterogeneous GPU serving, and black-box API contexts, highlighting ABC's practicality and impact for scalable AI deployments.

Abstract

Adaptive inference schemes reduce the cost of machine learning inference by assigning smaller models to easier examples, attempting to avoid invocation of larger models when possible. In this work we explore a simple, effective adaptive inference technique we term Agreement-Based Cascading (ABC). ABC builds a cascade of models of increasing size/complexity, and uses agreement between ensembles of models at each level of the cascade as a basis for data-dependent routing. Although ensemble execution introduces additional expense, we show that these costs can be easily offset in practice due to large expected differences in model sizes, parallel inference execution capabilities, and accuracy benefits of ensembling. We examine ABC theoretically and empirically in terms of these parameters, showing that the approach can reliably act as a drop-in replacement for existing models and surpass the best single model it aims to replace in terms of both efficiency and accuracy. Additionally, we explore the performance of ABC relative to existing cascading methods in three common scenarios: (1) edge-to-cloud inference, where ABC reduces communication costs by up to 14x; (2) cloud-based model serving, where it achieves a 3x reduction in rental costs; and (3) inference via model API services, where ABC achieves a 2-25x reduction in average price per token/request relative to state-of-the-art LLM cascades.

Figures (8)

• Figure 1: (a) Agreement-Based Cascading (ABC): ABC introduces a data-dependent routing scheme that uses agreement amongst an ensemble of models to determine whether to cascade to larger models. If the predictions of smaller ensembles do not align, the cascade moves to the next tier of larger models, continuing until agreement is reached or the largest model(s) are used. This can reduce cost by limiting the use of the largest models to cases where smaller models cannot reach consensus. (b)ABC is a natural baseline for adaptive inference due to (i) the vast number of pretrained models available to ML practitioners today; (ii) the fact that even small accuracy gains often require an order-of-magnitude increase in FLOPs, mirroring proposed scaling laws and resulting in large differences in model sizes between cascade tiers hestness2017deephenighan2020scalingmadaan_automix_2023. The pink dashed line represents the Pareto-optimal frontier, showing the models with the highest accuracy for a given computational budget. We show that ABC can effectively improve this frontier---allowing practitioners to achieve high accuracy without incurring the full computational cost of the largest models by invoking smaller models for 'easier' samples.
• Figure 2: Pareto curves of ABC vs. confidence-based cascades (WoC) wang2021wisdom and best single models on diverse tasks. For WoC, we tune its cascade configurations across the best four of its confidence thresholds and generate results from their most performant cascades. ABC maintains a Pareto-optimal curve, which consistently outperforms both methods in accuracy with lower FLOPs costs.
• Figure 3: Fraction of inference cost saved as a function of relative cost of models $(\gamma)$, assuming a fixed selection rate $\mathbb{P}(r(x) = 0)$. As parallelization decreases from fully parallel $(\rho=1)$ to sequential $(\rho=0)$, cost of evaluating ensembles increase and the cost savings decrease. When models across tiers are of similar size (e.g., smaller model is at most $5\times$ smaller, $\gamma \ge \frac{1}{5}$), some parallelization is needed for ABC to reduce costs effectively. However, for lower relative costs (e.g., smaller model is at least $50\times$ smaller, $\gamma \le \frac{1}{50}$), sequential and parallel settings achieve meaningful savings, showing ABC’s efficiency even with the added cost of using ensembles.
• Figure 4: (a) ABC for edge-to-cloud inference: We simulate a single-instance inference setup, as seen in real-time applications where predictions may need to be made as new data becomes available. ABC can enable small models to be served at the edge without sacrificing accuracy---leading to large savings in communication costs over the alternative of using only the highest accuracy/largest model residing in the cloud, or a single small and low-performing model on the edge. (b) Total GPU usage costs of ABC vs. using the best model. Agreement-Based Cascading, at reduced costs of GPU usage, exceeds the accuracy of the single best models in all task categories.
• Figure 5: Comparison of ABC against state-of-the-art cascade baselines for black-box API-based inference. The faded, hatched-patterned variants represent budget-friendly, 2-level cascade instances where we do not include the costly Tier 3. Most of these methods show competitive performance, but ABC matches their accuracy at significantly lower costs in all tasks. Note that all these methods (aside from the MoT-LLM cascade) incur additional setup costs not reflected in our plots.

Theorems & Definitions (3)

• Definition 4.1: Safe deferral rule
• Proposition 4.1
• Definition A.1: Admissible cascades