Proactive Routing to Interpretable Surrogates with Distribution-Free Safety Guarantees

Abstract

Model routing determines whether to use an accurate black-box model or a simpler surrogate that approximates it at lower cost or greater interpretability. In deployment settings, practitioners often wish to restrict surrogate use to inputs where its degradation relative to a reference model is controlled. We study proactive (input-based) routing, in which a lightweight gate selects the model before either runs, enabling distribution-free control of the fraction of routed inputs whose degradation exceeds a tolerance τ. The gate is trained to distinguish safe from unsafe inputs, and a routing threshold is chosen via Clopper-Pearson conformal calibration on a held-out set, guaranteeing that the routed-set violation rate is at most α with probability 1-δ. We derive a feasibility condition linking safe routing to the base safe rate π and risk budget α, along with sufficient AUC thresholds ensuring that feasible routing exists. Across 35 OpenML datasets and multiple black-box model families, gate-based conformal routing maintains controlled violation while achieving substantially higher coverage than regression conformal and naive baselines. We further show that probabilistic calibration primarily affects routing efficiency rather than distribution-free validity.

Figures (4)

• Figure 1: Feasibility landscape: critical AUC $\Phi_c(\pi, \alpha)$ as a function of base safe rate $\pi$ and risk budget $\alpha$. Darker regions require higher AUC for guaranteed feasibility. Stars mark our six $\tau$ operating points at $\alpha = 0.2$, colored by empirical outcome: green if routing achieves positive coverage, red if coverage is zero. At $\tau \leq 0$ (low $\pi$), the gate falls in the hard region and correctly abstains. At $\tau = 0.5$ and $\tau = 1.0$, routing succeeds despite AUC falling below $\Phi_c$, illustrating that the sufficient condition is conservative. At $\tau = 2.0$, AUC exceeds $\Phi_c$ and feasibility is guaranteed.
• Figure 2: Coverage vs. violation rate at $\alpha = 0.2$. Each point corresponds to a $\tau$ value averaged over 35 datasets. Gate conformal remains below $\alpha$. Regression conformal and naive operate above it.
• Figure 3: Left: gate ECE varies from 0.07 to 0.28 across $\tau$. Right: despite high ECE, conformal violation remains below $\alpha = 0.1$. Naive violations are 4--8$\times$ higher.
• Figure 4: Fraction of (dataset, $\tau$) pairs violating target $\alpha$ across the full $\alpha$ range. Gate conformal remains near the $\delta = 0.10$ line. Baselines converge only at very permissive $\alpha$.

Theorems & Definitions (16)

• Definition 1: Safe routing problem
• Proposition 1: Finite-sample guarantee
• Remark 1: Validity under threshold search
• Proposition 2: Feasibility condition
• Corollary 1: ROC Feasibility Geometry
• Theorem 1: Sufficient AUC for feasibility
• Remark 2
• Corollary 2
• Theorem 2: Tight AUC under concavity
• Theorem 3: Coverage bound