Learning To Defer To A Population With Limited Demonstrations

TL;DR

Learning To Defer To A Population With Limited Demonstrations tackles data scarcity in adaptive Learning-to-Defer (L2D) by introducing a context-aware semi-supervised framework that uses meta-learning to extract expert-specific embeddings from only a handful of demonstrations. The embeddings serve dual purposes: generating a large pseudo-labeled corpus for training a downstream L2D model and enabling on-the-fly adaptation to unseen experts at test time. The approach is validated across CIFAR-10, Fashion-MNIST, and GTSRB, showing rapid convergence toward oracle-level performance with few annotations and strong generalization to new experts. This work lowers the annotation burden for population-aware L2D, enabling scalable human-AI collaboration, and provides code and configurations to support reproducibility.

Abstract

This paper addresses the critical data scarcity that hinders the practical deployment of learning to defer (L2D) systems to the population. We introduce a context-aware, semi-supervised framework that uses meta-learning to generate expert-specific embeddings from only a few demonstrations. We demonstrate the efficacy of a dual-purpose mechanism, where these embeddings are used first to generate a large corpus of pseudo-labels for training, and subsequently to enable on-the-fly adaptation to new experts at test-time. The experiment results on three different datasets confirm that a model trained on these synthetic labels rapidly approaches oracle-level performance, validating the data efficiency of our approach. By resolving a key training bottleneck, this work makes adaptive L2D systems more practical and scalable, paving the way for human-AI collaboration in real-world environments. To facilitate reproducibility and address implementation details not covered in the main text, we provide our source code and training configurations at https://github.com/nil123532/learning-to-defer-to-a-population-with-limited-demonstrations.

Figures (2)

• Figure 1: A small context set $\mathcal{C}_{e}= \{(x_b,y_b,h_b^{e})\}_{b=1}^{B}$ summarises expert $e$. Image features from the frozen backbone $\Phi_{\text{emb}}$ are concatenated with label embeddings and processed by the set encoder $\Phi_{\text{enc}}$, yielding the expert embedding $\boldsymbol{\psi}_{e}$. For a query image $x_j$ the same $\Phi_{\text{emb}}$ encodes the image; this vector is combined with $\boldsymbol{\psi}_{e}$ and passed to the expert-aware head $\Phi_{\text{ex}}$, which predicts whether the expert will label the query correctly or incorrectly. Each expert contributes a supervised loss $\mathcal{L}_{s}^{e}$ on its labelled data and a consistency loss $\mathcal{L}_{u}^{e}$ on its unlabelled data;averaging across these terms yields the meta-objective $\mathcal{L}$ that trains the entire multi-task model across all experts.
• Figure 2: Impact of increasing $L$ on downstream L2D-Pop performance. Each column corresponds to one dataset—Fashion-MNIST (left), CIFAR-10 (center), and GTSRB (right). Top row: overall human–AI system accuracy. Solid lines show the four model variants (NP + Attention, NP, Finetune, and Single); matching-color dashed lines denote their oracle upper bounds, and the black dashed line marks the classifier-alone baseline. Bottom row: expert accuracy after deferral. Solid lines again show the four variants with their oracle upper bounds (matching-color dashed lines); the grey dashed line indicates the expert-alone baseline.