Online Decision Deferral under Budget Constraints

TL;DR

This work addresses online decision deferral under budget constraints by framing it as a two-armed contextual bandit with context-dependent rewards for a human and an ML model. It advances a generalized linear bandit algorithm with optimistic parameter estimates, plus a neural-linear extension that learns context embeddings, and provides regret guarantees relative to an optimal static policy under budget $B$. The authors demonstrate both theoretical guarantees and strong empirical performance on synthetic data and real tasks (knapsack and ImageNet16H), showing the approach can adapt to distribution shifts and balance deferral costs with model performance. The framework enables effective human-in-the-loop decisions in resource-constrained environments, with practical impact for domains where expert time is limited and data distributions evolve over time.

Abstract

Machine Learning (ML) models are increasingly used to support or substitute decision making. In applications where skilled experts are a limited resource, it is crucial to reduce their burden and automate decisions when the performance of an ML model is at least of equal quality. However, models are often pre-trained and fixed, while tasks arrive sequentially and their distribution may shift. In that case, the respective performance of the decision makers may change, and the deferral algorithm must remain adaptive. We propose a contextual bandit model of this online decision making problem. Our framework includes budget constraints and different types of partial feedback models. Beyond the theoretical guarantees of our algorithm, we propose efficient extensions that achieve remarkable performance on real-world datasets.

Figures (11)

• Figure 1: The deferral learner observes inputs online over a time horizon $T$, then decides whether to pay $c_t$ and defer to the human or to let an ML model decide for $c_t=0$. The deferral learner achieves the reward of the human decision $r_{\text{h},t} = r_\text{h}(x_t)$ if the context was deferred to the human, and the ml decision reward $r_{\text{m},t} = r_\text{m}(x_t)$ otherwise. The observation $O_t$ is the reward of the decision $r_{\pi(x_t),t}$ in the pure bandit feedback setting. In the full information setting, $O_t = \lbrace r_{\text{m},t}, r_{\pi(x_t),t} \rbrace$.
• Figure 2: Mean and standard deviation of the regret over 100 trials. The reward and cost functions are sampled uniformly at random from $[0,1]^d$ for each trial.The algorithm is run over $T=50000$ random contexts with $B=8000$. Then, the reward received by OPT is computed for the same contexts.
• Figure 3: Performance of Algorithm \ref{['alg:glm']} as a percentage of OPT across five different budgets. The two plots correspond to two different human/model reward functions. (Left) $\theta_h\sim \{x \in \{0,1\}^{20} : \lVert x\rVert_1\} = 10\}$, and $\theta_\text{m} = 1-\theta_\text{h}$. (Right) $\theta_\text{h}\sim [0,1]^{20}$ and $\theta_\text{m}\sim[0,0.5]^{20}$. In both, the cost function is $w_t\sim[0,1]^{20}$. Each experiment runs for $T=50000$ time steps, and the mean and standard deviation over 20 trials are shown.
• Figure 4: The reward over time of the NeuralLinear and Linear variants on the Knapsack Problem dataset with no budget constraints. The reward of always deferring to the human (HumanOnly) and the model (ModelOnly) are plotted for comparison.
• Figure 5: Loss versus OPT of the NeuralLinear and Linear variants of the algorithm on the Knapsack Problem dateset with three different budget constraints: infinite budget (Left), $B=1/2T$ (Middle), and $B=1/4T$ (Right). The lines show the mean over 20 trials (rearranging the order of the participants), and the shaded region represents one standard deviation.

Theorems & Definitions (14)

• Definition 3.1: Optimal Static Policy
• Definition 4.1: Maximum Likelihood Estimator
• Definition 4.2: Optimistic estimates
• Corollary 4.3: based on agrawal2016linear
• Lemma A.1: Lemma 3 of li2017provably
• Lemma A.2: Lemma 2 of li2017provably
• Corollary A.3: Corollary 1 of agrawal2016linear
• proof
• Corollary A.4: Corollary 2 of agrawal2016linear
• proof