Conditional Performance Guarantee for Large Reasoning Models

TL;DR

The paper tackles the challenge of providing reliable efficiency guarantees for large reasoning models by moving beyond marginal PAC bounds to group-conditional guarantees. It introduces Group PAC (G-PAC) reasoning for known groupings and Clustered PAC (C-PAC) reasoning for unknown groupings, both achieving group-level risk control via per-group calibration of uncertainty thresholds and upper confidence bounds. Theoretical results establish group-wise validity, optimality of oracle partitions, and bounds on coverage gaps when groupings are learned, with experiments across MATH-500, ZebraLogic, GPQA, and Arena-Hard showing substantial runtime savings while maintaining group-specific error guarantees. The framework demonstrates that exploiting heterogeneity through grouping can strictly improve efficiency without sacrificing reliability, providing a practical pathway to scalable, trustworthy reasoning in heterogeneous data regimes.

Abstract

Large reasoning models have shown strong performance through extended chain-of-thought reasoning, yet their computational cost remains significant. Probably approximately correct (PAC) reasoning provides statistical guarantees for efficient reasoning by adaptively switching between thinking and non-thinking models, but the guarantee holds only in the marginal case and does not provide exact conditional coverage. We propose G-PAC reasoning, a practical framework that provides PAC-style guarantees at the group level by partitioning the input space. We develop two instantiations: Group PAC (G-PAC) reasoning for known group structures and Clustered PAC (C-PAC) reasoning for unknown groupings. We prove that both G-PAC and C-PAC achieve group-conditional risk control, and that grouping can strictly improve efficiency over marginal PAC reasoning in heterogeneous settings. Our experiments on diverse reasoning benchmarks demonstrate that G-PAC and C-PAC successfully achieve group-conditional risk control while maintaining substantial computational savings.

Figures (7)

• Figure 1: Trade-off between feasibility and efficiency. Left: PAC reasoning uses a single threshold for all inputs, which is feasible but not the most efficient. Right: conditional PAC reasoning requires per-input guarantees, which is fully efficient but impossible. Middle: G-PAC reasoning balances this trade-off by grouping similar inputs and calibrating dynamic thresholds $\widehat{u}_j$ for each group, making it both feasible and group-conditional efficient.
• Figure 2: G-PAC controls the group-conditional performance loss below the target while vanilla PAC fails (dark blue) across three different uncertainty scores. All results are obtained using the Qwen model. The figure reports the overall and per-category performance losses on three reasoning benchmarks.
• Figure 3: More calibration samples enhance efficiency of reasoning. Experimental results of G-PAC reasoning for different calibration ratios on MATH-500. The red dashed line $\varepsilon$ means the target risk level.
• Figure 4: The category distribution for three verfiable benchmarks.
• Figure 5: Expected Calibration Error across three verifiable benchmarks using Qwen LLMs.

Theorems & Definitions (17)

• Remark 1
• Definition 2.1: PAC-efficient
• Definition 3.1: Group-conditional PAC efficiency
• Remark 2
• Theorem 4.2: PAC guarantee for known grouping
• Theorem 4.3: PAC finite-sample guarantee for known grouping
• Remark 3
• Definition 4.4: Oracle optimal partition
• Remark 4: Efficiency as non-thinking model usage
• Proposition 4.5: Group-conditional benefit