Causal Bandits: The Pareto Optimal Frontier of Adaptivity, a Reduction to Linear Bandits, and Limitations around Unknown Marginals

TL;DR

The paper studies adaptivity to causal structure in bandits with post-action context by defining conditionally benign environments and characterizing the Pareto frontier of achievable regret across benign and arbitrary settings. It introduces dynamic balancing to combine a benign-focused learner (C-UCB) with a robust, model-free learner (UCB), yielding optimal tradeoffs and a quantified price of adaptivity. A novel reduction from causal bandits to linear bandits enables the first instance-dependent regret bounds in the benign regime via phased elimination, with the key complexity term $d_{ u}$ capturing intrinsic problem structure. The work also demonstrates that knowing the marginals is generally necessary for improving minimax rates, though partial marginal knowledge can suffice in some regimes, and discusses open questions around tighter instance-dependent theories and simpler adaptive strategies.

Abstract

In this work, we investigate the problem of adapting to the presence or absence of causal structure in multi-armed bandit problems. In addition to the usual reward signal, we assume the learner has access to additional variables, observed in each round after acting. When these variables $d$-separate the action from the reward, existing work in causal bandits demonstrates that one can achieve strictly better (minimax) rates of regret (Lu et al., 2020). Our goal is to adapt to this favorable "conditionally benign" structure, if it is present in the environment, while simultaneously recovering worst-case minimax regret, if it is not. Notably, the learner has no prior knowledge of whether the favorable structure holds. In this paper, we establish the Pareto optimal frontier of adaptive rates. We prove upper and matching lower bounds on the possible trade-offs in the performance of learning in conditionally benign and arbitrary environments, resolving an open question raised by Bilodeau et al. (2022). Furthermore, we are the first to obtain instance-dependent bounds for causal bandits, by reducing the problem to the linear bandit setting. Finally, we examine the common assumption that the marginal distributions of the post-action contexts are known and show that a nontrivial estimate is necessary for better-than-worst-case minimax rates.

Figures (1)

• Figure 1: The Pareto-optimal frontier of simultaneously achievable rates of regret in (left axis) the class of conditionally benign environments and (bottom axis) the class of all environments. Shaded regions are unobtainable. All rates are determined up to log terms. Among algorithms that achieve minimax rates on conditionally benign environments, the previously best known algorithm (HAC-UCB) is dominated by an instance of Dynamic Balancing, which our results also demonstrate is Pareto optimal.

Theorems & Definitions (42)

• Definition 2.1
• Remark 2.2
• Definition 3.1
• Theorem 3.2
• Definition 3.3
• Proposition 3.4
• Theorem 3.5
• Corollary 3.6
• Theorem 3.7
• Theorem 4.1: Worst-case regret bound for $\mathop{\mathrm{PE}}\nolimits$