A note on the impossibility of conditional PAC-efficient reasoning in large language models
Hao Zeng
TL;DR
The paper addresses whether conditional PAC-efficient reasoning is possible for large language-model routing. It formalizes a router-based composite predictor that switches between an expensive expert and a cheaper fast model and defines the pointwise risk. The main result shows that, under distribution-free assumptions and non-atomic input spaces, achieving conditional PAC efficiency forces the routing to defer to the expert with probability at least 1-α for almost every input, yielding no practical speedups. This extends known marginal PAC guarantees and aligns with impossibility results in conformal prediction, guiding practitioners to prioritize marginal guarantees or adopt distributional assumptions for conditional notions.
Abstract
We prove an impossibility result for conditional Probably Approximately Correct (PAC)-efficient reasoning in large language models. While recent work has established marginal PAC efficiency guarantees for composite models that switch between expensive expert models and cheaper fast models, we show that conditional (pointwise) guarantees are impossible in the distribution-free setting. Specifically, for non-atomic input spaces, any algorithm achieving conditional PAC efficiency must be trivial in the sense that it defers to the expert model with probability at least $1-α$ for almost every input.