C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning

TL;DR

C3PO tackles the high inference cost of large language models by learning a label-free cascade of models with early-exit thresholds that approximate the strongest model's accuracy while meeting a probabilistic cost budget. It combines self-supervised threshold learning on unlabeled prompts, conformal prediction to bound test-time cost overruns, and PAC-Bayes guarantees to bound generalization error. The approach yields state-of-the-art cost-efficiency across 16 diverse reasoning benchmarks and demonstrates robustness to distribution shift and model-family composition. This work enables scalable, sustainable deployment of LLM reasoning by providing principled control over computation without requiring ground-truth labels for training cascade decisions.

Abstract

Large language models (LLMs) have achieved impressive results on complex reasoning tasks, but their high inference cost remains a major barrier to real-world deployment. A promising solution is to use cascaded inference, where small, cheap models handle easy queries, and only the hardest examples are escalated to more powerful models. However, existing cascade methods typically rely on supervised training with labeled data, offer no theoretical generalization guarantees, and provide limited control over test-time computational cost. We introduce C3PO (Cost Controlled Cascaded Prediction Optimization), a self-supervised framework for optimizing LLM cascades under probabilistic cost constraints. By focusing on minimizing regret with respect to the most powerful model (MPM), C3PO avoids the need for labeled data by constructing a cascade using only unlabeled model outputs. It leverages conformal prediction to bound the probability that inference cost exceeds a user-specified budget. We provide theoretical guarantees on both cost control and generalization error, and show that our optimization procedure is effective even with small calibration sets. Empirically, C3PO achieves state-of-the-art performance across a diverse set of reasoning benchmarks including GSM8K, MATH-500, BigBench-Hard and AIME, outperforming strong LLM cascading baselines in both accuracy and cost-efficiency. Our results demonstrate that principled, label-free cascade optimization can enable scalable LLM deployment.

Figures (28)

• Figure 1: Left: Existing LLM cascading strategies include supervised learning, reinforcement learning, and learning-free heuristics. In contrast, C3PO proposes a novel, cost-constrained, self supervised (label-free) cascade paradigm. Right: Surpassing existing cascade approaches tremendously, C3PO offers markedly superior cost-effectiveness across 16 benchmarks, requiring less than 20% of the cost of the most powerful model (MPM) (cost shown in purple) for an accuracy gap of at most 2, 5, and 10% using a LLAMA cascade. In this boxplot, each dot represents a dataset and the whiskers extend to 90% coverage.
• Figure 2: C3PO achieves the best performance for a wide range of cost budgets on the LLAMA cascade. Error bars denote 90% confidence interval. GPT and QWEN cascade results in App. \ref{['app:full_results']}.
• Figure 3: Average accuracies and dollar costs of different algorithms for five different difficulty levels of the MATH-500 dataset using the LLAMA cascade.
• Figure 4: Left: Results of GPT cascade on AIME. Right: Here we see that a mixed model family cascade comprised of LLaMA 3.2 1B-Instruct, Qwen 2.5 32B-Instruct and GPT 4o-mini operates with roughly the same performance as cascades of models comprised within the same family. Therefore, we conclude that the method is robust to LLM family selection.
• Figure 5: Distribution shift experiment. Training on GSM8K, testing on MATH500 shows that C3PO has the best domain adaptation performance.

Theorems & Definitions (10)

• Theorem 1: Conformal Cost Guarantee
• proof
• Theorem 2: Generalization Bound
• proof
• Theorem 1: Conformal Cost Guarantee
• proof
• Theorem 2: Generalization Bounds
• proof
• Theorem 3: Upper Bound on the MDC in Empirical Regret
• proof