Contextual Bandits with Budgeted Information Reveal

TL;DR

This work introduces a novel optimization and learning algorithm that effectively combines the strengths of two algorithmic approaches in a seamless manner, including an online primal-dual algorithm for deciding the optimal timing to reach out to patients and a contextual bandit learning algorithm to deliver personalized treatment to the patient.

Abstract

Contextual bandit algorithms are commonly used in digital health to recommend personalized treatments. However, to ensure the effectiveness of the treatments, patients are often requested to take actions that have no immediate benefit to them, which we refer to as pro-treatment actions. In practice, clinicians have a limited budget to encourage patients to take these actions and collect additional information. We introduce a novel optimization and learning algorithm to address this problem. This algorithm effectively combines the strengths of two algorithmic approaches in a seamless manner, including 1) an online primal-dual algorithm for deciding the optimal timing to reach out to patients, and 2) a contextual bandit learning algorithm to deliver personalized treatment to the patient. We prove that this algorithm admits a sub-linear regret bound. We illustrate the usefulness of this algorithm on both synthetic and real-world data.

Figures (8)

• Figure 1: A depiction of the problem breakdown in the electronic toothbrush application, which we extensively investigate in our experiments. In our problem reformulation, it is equally valid to view the recommender as residing on the intervention-delivery device.
• Figure 2: Regret comparison under known $\mathbf{p}^*$ with $B=10$. Left: cumulative regret averaged over 50 instances (each with a unique $\theta_*$ value) each with 50 replications. Middle and right: regret comparison between PD2-UCB, Naive-UCB, and PD1-UCB, at $T=300$; each dot represents one instance averaged over 50 replications.
• Figure 3: Average reward comparison on the ROBAS3 dataset. The y-axis is the mean and $\pm 1.96 \cdot$ standard error of the average user rewards $\left(\bar{R}=\frac{1}{10} \sum_{i=1}^{10} \frac{1}{t_0} \sum_{s=1}^{t_0} R_{i, s}\right)$ for decision times $t_0 \in[20,40,60,80,100,120,140]$ across $50$ experiments and 10 users. Standard error is $\frac{\sum_i^{10}\hat{\sigma}_i}{10\sqrt{50}}$ where $\hat{\sigma}_i$ is the user-specific standard error.
• Figure G.1: Average regret (left) and scatter plot for $B=20$ at $T=300$ under known $\mathbf{p}^*$. Each dot corresponds to one instance averaged over 50 replications.
• Figure G.2: Average regret (left) and scatter plots for $B=30$ at $T=300$ under known $\mathbf{p}^*$. Each dot corresponds to one instance averaged over 50 replications.

Theorems & Definitions (26)

• Remark 1: Objective function of \ref{['pblm: clairvoyant']}
• Proposition 1
• Corollary 1
• Remark 2: Existence of $c_{\max}$
• Proposition 2
• Theorem 1
• Remark 3: Regret bound
• Proposition C.1
• Proposition C.1
• proof