Expert-Agnostic Learning to Defer

TL;DR

This work tackles the challenge of Learning to Defer (L2D) under unseen experts by introducing Expert-Agnostic Learning to Defer (EA-L2D), a Bayesian framework that builds explicit per-class expert behavioural representations using a Beta-Binomial model. The deferral decision is made by an expert-agnostic rejector that operates on classifier confidence and the expert's quantified competence, enabling robust generalisation to OOD experts and multiple expertise patterns. The approach supports incorporating prior knowledge through informative priors and reduces annotation costs by using a surrogate loss with uncertainty-aware weighting, with theoretical guarantees and empirical validation on four medical-imaging datasets. Across both ID and OOD scenarios, EA-L2D achieves up to 28% relative improvements in deferral performance and demonstrates stronger robustness to distribution shifts than L2D-Pop, highlighting its practical relevance for safe, collaborative AI in high-stakes domains.

Abstract

Learning to Defer (L2D) trains autonomous systems to handle straightforward cases while deferring uncertain ones to human experts. Recent advancements in this field have introduced methods that offer flexibility to unseen experts at test time. However, we find these approaches struggle to generalise to experts with behaviours not seen during training, require extensive human annotation, and lack mechanisms for incorporating prior knowledge of expert capabilities. To address these challenges, we introduce Expert-Agnostic Learning to Defer (EA-L2D), a novel L2D framework that employs a Bayesian approach to model expert behaviour in an \textit{expert-agnostic} fashion. Across benchmark medical imaging datasets (HAM10000, Blood Cells, Retinal OCT, and Liver Tumours), EA-L2D significantly outperforms prior methods on unseen experts, achieving up to a 28\% relative improvement, while also matching or exceeding state-of-the-art performance on seen experts.

Figures (8)

• Figure 1: Overview of Expert-Agnostic Learning to Defer (EA-L2D). An arbitrary expert $E$'s context data $\mathcal{D}_C^E$ is modelled (Beta-Binomial, §\ref{['rep']}) into a behavioural representation $\mathcal{R}^E$ (class-wise posterior means/variances), with optional priors (§\ref{['sec:priors']}). EA-L2D's expert-agnostic rejector (§\ref{['def_logit']}) compares classifier outputs against $\mathcal{R}^E$. This 'agnosticism' (operating without dependence on specific expert identities or known specialisms) enables robust generalisation to diverse and unseen experts, a key advantage over methods that learn profiles tied to training-time experts.
• Figure 2: Example rejector inputs/outputs. Expert A: Classifier uncertain, expert highly competent $\Rightarrow$ negative $g_{\perp}$ (no deferral). Expert B: Classifier confident and prediction aligns with expert’s expertise $\Rightarrow$ positive $g_{\perp}$ (deferral).
• Figure 3: Performance comparison between EA-L2D and L2D-Pop using a single in-distribution (ID) and out-of-distribution (OOD) expert, across varying numbers of expertise classes ($E^*$), on Liver Tumours and Blood Cells datasets. Asterisks denote statistical significance in favour of EA-L2D: * for $p<0.1$ and ** for $p<0.05$ (via two-tailed paired samples t-test).
• Figure 4: Effect of prior knowledge on EA-L2D performance (AURSAC) for a Class 3 HAM10000 expert.
• Figure 5: Latent behavioural representations learned by L2D-Pop on Retinal OCT, visualised by (a) predicted class, (b) expert identity, and (c) deferral confidence. Outlined circles indicate OOD experts with test images; faded, non-outlined faded circles represent training images from ID experts.

Theorems & Definitions (19)

• Proposition 1
• Proposition 2: Sample Complexity for Overcoming Prior Misspecification
• Proposition 3
• Proposition \ref{prop:eal2d}
• proof
• Remark A.1: Practical Robustness of Expert-agnosticism
• Theorem C.1: Hoeffding's Inequality
• Theorem C.2: Boole's Inequality
• Proposition \ref{prop2}
• proof