Theory and Algorithms for Learning with Multi-Class Abstention and Multi-Expert Deferral

TL;DR

This work develops a unified theory and algorithmic framework for learning with multi-class abstention and multi-expert deferral. It introduces score-based and predictor-rejector formulations, plus single-stage and two-stage schemes, and proves strong, non-asymptotic ${\mathscr H}$-consistency guarantees using minimizability-gap-aware surrogate losses. The theory covers both classification and regression, and extends to regression with multiple experts by constructing novel surrogate losses with realizable ${\mathscr H}$-consistency. Empirically, the proposed surrogates demonstrate gains over prior methods on standard benchmarks (CIFAR-10/100, SVHN) and illustrate practical benefits when deferring to multiple, potentially costly, experts. The results provide principled foundations for abstention/deferral in diverse settings and offer concrete algorithms with finite-sample guarantees and realizability properties.

Abstract

Large language models (LLMs) have achieved remarkable performance but face critical challenges: hallucinations and high inference costs. Leveraging multiple experts offers a solution: deferring uncertain inputs to more capable experts improves reliability, while routing simpler queries to smaller, distilled models enhances efficiency. This motivates the problem of learning with multiple-expert deferral. This thesis presents a comprehensive study of this problem and the related problem of learning with abstention, supported by strong consistency guarantees. First, for learning with abstention (a special case of deferral), we analyze score-based and predictor-rejector formulations in multi-class classification. We introduce new families of surrogate losses and prove strong non-asymptotic, hypothesis set-specific consistency guarantees, resolving two existing open questions. We analyze both single-stage and practical two-stage settings, with experiments on CIFAR-10, CIFAR-100, and SVHN demonstrating the superior performance of our algorithms. Second, we address general multi-expert deferral in classification. We design new surrogate losses for both single-stage and two-stage scenarios and prove they benefit from strong $H$-consistency bounds. For the two-stage scenario, we show that our surrogate losses are realizable $H$-consistent for constant cost functions, leading to effective new algorithms. Finally, we introduce a novel framework for regression with deferral to address continuous label spaces. Our versatile framework accommodates multiple experts and various cost structures, supporting both single-stage and two-stage methods. It subsumes recent work on regression with abstention. We propose new surrogate losses with proven $H$-consistency and demonstrate the empirical effectiveness of the resulting algorithms.

Figures (2)

• Figure 1: Counterexample for score-based abstention losses.
• Figure 2: Illustration of the scenario of learning with multiple-expert deferral ($n=3$ and ${n_e}=2$).

Theorems & Definitions (111)

• Theorem 1.1: $\sH$-consistency bounds for score-based surrogates
• Theorem 1.2: Characterization of minimizability gaps
• Theorem 1.3
• Theorem 1.4: $\sH$-consistency bounds for two-stage surrogates
• Theorem 2.1: Negative result for single-stage surrogates
• Theorem 2.2: $(\sH, \sR)$-consistency bounds for single-stage surrogates
• Corollary 2.2: Excess error bounds for single-stage surrogates
• Theorem 2.3: $\sR$-consistency bounds for second-stage surrogates
• Corollary 2.4
• Theorem 2.5: $(\sH, \sR)$-consistency bounds for two-stage approach