Causal Contextual Bandits with Adaptive Context

TL;DR

This work investigates causal contextual bandits with adaptive contexts, where the environment selects a stochastic context after an initial learner action to maximize reward. The authors introduce ConvExplore, a convex-optimization-based exploration strategy that estimates start-state transitions, context-specific causal parameters, and context rewards, then computes a near-optimal policy. A key contribution is the instance-dependent parameter $\lambda$, which governs simple-regret bounds, and a matching lower bound demonstrating tightness for a broad class of instances. Experiments with realistic causal graphs corroborate theoretical findings and show substantial improvements over uniform exploration, with code publicly available for replication.

Abstract

We study a variant of causal contextual bandits where the context is chosen based on an initial intervention chosen by the learner. At the beginning of each round, the learner selects an initial action, depending on which a stochastic context is revealed by the environment. Following this, the learner then selects a final action and receives a reward. Given $T$ rounds of interactions with the environment, the objective of the learner is to learn a policy (of selecting the initial and the final action) with maximum expected reward. In this paper we study the specific situation where every action corresponds to intervening on a node in some known causal graph. We extend prior work from the deterministic context setting to obtain simple regret minimization guarantees. This is achieved through an instance-dependent causal parameter, $λ$, which characterizes our upper bound. Furthermore, we prove that our simple regret is essentially tight for a large class of instances. A key feature of our work is that we use convex optimization to address the bandit exploration problem. We also conduct experiments to validate our theoretical results, and release our code at our project GitHub repository: https://github.com/adaptiveContextualCausalBandits/aCCB.

Figures (10)

• Figure 1: Flowchart illustrating the decision-making process of an advertiser posting ads on a platform like Amazon, and the subsequent interaction with the platform.
• Figure 2: The transition to a particular context (chosen context in the figure on the left) is decided by the environment, whereas the interventions at the start state and an intermediate context (chosen interventions in the figure on the right) are chosen by the learner.
• Figure 3: We plot the Simple Regret under $\textsc{ConvExplore}$ and $\textsc{UnifExplore}$. The figure on the left (\ref{['figure: experimental results']}a) plots expected simple regret vs time, for the setup $n=25$, $k=25$, $\lambda=50$, $\varepsilon = 0.3$ and $m=2$ for all contexts. The figure on the right (\ref{['figure: experimental results']}b) plots expected simple regret with $\lambda$. It was performed with the parameters: $T=25000$, $k=25$, $m_0 = 2$ and $\varepsilon = 0.3$.
• Figure 4: We plot various baselines for two metrics of interest (1) Probability of the algorithm finding the best interventions and (2) Simple regret. These plots illustrate how these metrics vary with the exploration budget.
• Figure 5: We plot the variation of probability of finding the best intervention and simple regret with the number of contexts. Notice the outperformance of $\textsc{ConvExplore}$ vs. the other baselines.

Theorems & Definitions (49)

• Theorem 1
• Theorem 2
• Theorem
• Definition 1
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof