Adaptive Experimentation When You Can't Experiment

TL;DR

This paper considers a more general underlying model captured by a linear structural equation and forms pure exploration linear bandits in this setting, believed to be the first work considering a setting where noise is confounded.

Abstract

This paper introduces the \emph{confounded pure exploration transductive linear bandit} (\texttt{CPET-LB}) problem. As a motivating example, often online services cannot directly assign users to specific control or treatment experiences either for business or practical reasons. In these settings, naively comparing treatment and control groups that may result from self-selection can lead to biased estimates of underlying treatment effects. Instead, online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment. Our methodology provides online services with an adaptive experimental design approach for learning the best-performing treatment for such \textit{encouragement designs}. We consider a more general underlying model captured by a linear structural equation and formulate pure exploration linear bandits in this setting. Though pure exploration has been extensively studied in standard adaptive experimental design settings, we believe this is the first work considering a setting where noise is confounded. Elimination-style algorithms using experimental design methods in combination with a novel finite-time confidence interval on an instrumental variable style estimator are presented with sample complexity upper bounds nearly matching a minimax lower bound. Finally, experiments are conducted that demonstrate the efficacy of our approach.

Figures (6)

• Figure 1: Causal graph of the model.
• Figure 2: (a) A bar chart showing $\mathbb{E}[y|x=w]=w^{\top}\theta$ and $\mathbb{E}[y|z=w]=z^{\top}\Gamma\theta$ for all $w\in \mathcal{W}$. This chart shows that the optimal evaluation vector is $w^{\ast}=e_1=\mathop{\mathrm{arg\,max}}\limits_{w\in \mathcal{W}}\mathbb{E}[y|x=w]$, while $e_6=\mathop{\mathrm{arg\,max}}\limits_{w\in \mathcal{W}}\mathbb{E}[y|z=w]$ and consequently estimation based on this quantity is problematic. (b) The probability of identifying $w^{\ast}=e_1$ for a collection of algorithms on the CPET-LB instance from Section \ref{['sec:motivating']}.
• Figure 3: (a) A visual depiction of the problem instance from Section \ref{['sec:illustrative']}. The user is presented with encouragement $I_t\in \mathcal{A}$ and the user choice is given by $J_t$ where $J_t = \min_{j\in \mathcal{A}}|I_t + u_t - j|$ and $u_t \sim \mathcal{N}(0,\sigma^2_u)$. b) A heat-map showing the structural parameter $\Gamma$ for the problem instance from Section \ref{['sec:illustrative']}. (c) A bar chart showing $\mathbb{E}[y|x=w]=w^{\top}\theta$ and $\mathbb{E}[y|z=w]=z^{\top}\Gamma\theta$ for all $w\in \mathcal{W}$. This chart shows that the optimal evaluation vector is $w^{\ast}=e_1=\mathop{\mathrm{arg\,max}}\limits_{w\in \mathcal{W}}\mathbb{E}[y|x=w]$, while $e_6=\mathop{\mathrm{arg\,max}}\limits_{w\in \mathcal{W}}\mathbb{E}[y|z=w]$ and consequently estimation based on this quantity is problematic. (d) The probability of identifying $w^{\ast}=e_1$ for a collection of algorithms on the CPET-LB instance described in Section \ref{['sec:illustrative']}. Standard optimistic sampling approaches in combination with an ordinary least squares estimator leads to faulty inferences. Given an instrumental variable estimator, these experimental designs eventually give high probability identification but do so inefficiently compared to our proposed approach (see Section \ref{['sec:algorithms']}).
• Figure 4: Sample complexity for algorithms on CPET-LB problems. Our approach is consistently competitive across the experiments.
• Figure : CPEG:Confounded pure exploration with $\Gamma$

Theorems & Definitions (73)

• Definition 1.1: $\delta$-PAC
• Lemma 2.1
• Lemma 2.2
• Theorem 2.3
• Remark 2.4
• Remark 2.5
• Theorem 3.1
• Lemma 3.2
• Remark 3.3
• Theorem 3.4