OCCAM: Towards Cost-Efficient and Accuracy-Aware Classification Inference

TL;DR

OCCAM tackles cost-aware classification by formulating an optimal model portfolio problem: assign queries to a pool of pre-trained classifiers to maximize overall accuracy under a user-specified budget $B$. It introduces a data-driven, unbiased low-variance accuracy estimator based on a nearest-neighbor approach under the $r$-separation assumption, and solves the resulting assignment with an integer linear program regularized to mitigate estimation risk. The method is evaluated on multiple image-classification benchmarks, showing up to $40\%$ cost reductions with negligible accuracy loss and favorable model-usage patterns across budgets. The work provides a training-free alternative to router-based or MoE approaches, with practical implications for cost-efficient and accuracy-aware inference in real-world services.

Abstract

Classification tasks play a fundamental role in various applications, spanning domains such as healthcare, natural language processing and computer vision. With the growing popularity and capacity of machine learning models, people can easily access trained classifiers as a service online or offline. However, model use comes with a cost and classifiers of higher capacity (such as large foundation models) usually incur higher inference costs. To harness the respective strengths of different classifiers, we propose a principled approach, OCCAM, to compute the best classifier assignment strategy over classification queries (termed as the optimal model portfolio) so that the aggregated accuracy is maximized, under user-specified cost budgets. Our approach uses an unbiased and low-variance accuracy estimator and effectively computes the optimal solution by solving an integer linear programming problem. On a variety of real-world datasets, OCCAM achieves 40% cost reduction with little to no accuracy drop.

Figures (15)

• Figure 1: We investigate Tiny ImageNet dataset consisting of $200$ classes (see \ref{['sec:eval']} for details). We observe that (a) smaller classifiers (e.g., ResNet-18) generally yield lower accuracy, (b) small classifiers can agree with large classifiers at a high frequency (each entry indicates the percentage of queries on which the classifier on the row makes the right prediction and so does the classifier on the column), (c) our approach OCCAM achieves $20\%$ cost reduction with less than $1\%$ accuracy drop.
• Figure 2: OCCAM: Optimization with Cost Constraints towards Accuracy Maximization.
• Figure 3: Accuracy-cost tradeoffs achieved by OCCAM and baselines, for different cost budgets.
• Figure 4: (a) Model usage breakdown by OCCAM under different cost budgets, (b) OCCAM accuracy with different sample sizes, and (c) OCCAM performance with different $\lambda$ choices
• Figure 5: Intuitive example of well separated images from Tiny ImageNet under $l_{\infty}$ distance metric using ResNet-18 features. Images from different classes (e.g., "goldfish" and "bullfrog") are typically well separated by a non-zero distance.

Theorems & Definitions (12)

• Definition 3.1: Model Portfolio
• Definition 3.2: Optimal Model Portfolio
• Definition 4.1: $r$-separation yang2020closer
• Definition 4.2: Lipschitz Continuity
• Lemma 4.3
• Lemma 4.4
• Lemma 4.5: Asymptotically Unbiased Estimator
• Lemma 4.6: Asymptotically Low-Variance Estimator
• proof
• proof