Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Partial Structure Discovery is Sufficient for No-regret Learning in Causal Bandits

Muhammad Qasim Elahi, Mahsa Ghasemi, Murat Kocaoglu

TL;DR

This work identifies that discovering the full causal structure is unnecessary; however, no existing work provides the necessary and sufficient components of the causal graph, and formally characterize the set of necessary and sufficient latent confounders one needs to detect or learn to ensure that all possibly optimal arms are identified correctly.

Abstract

Causal knowledge about the relationships among decision variables and a reward variable in a bandit setting can accelerate the learning of an optimal decision. Current works often assume the causal graph is known, which may not always be available a priori. Motivated by this challenge, we focus on the causal bandit problem in scenarios where the underlying causal graph is unknown and may include latent confounders. While intervention on the parents of the reward node is optimal in the absence of latent confounders, this is not necessarily the case in general. Instead, one must consider a set of possibly optimal arms/interventions, each being a special subset of the ancestors of the reward node, making causal discovery beyond the parents of the reward node essential. For regret minimization, we identify that discovering the full causal structure is unnecessary; however, no existing work provides the necessary and sufficient components of the causal graph. We formally characterize the set of necessary and sufficient latent confounders one needs to detect or learn to ensure that all possibly optimal arms are identified correctly. We also propose a randomized algorithm for learning the causal graph with a limited number of samples, providing a sample complexity guarantee for any desired confidence level. In the causal bandit setup, we propose a two-stage approach. In the first stage, we learn the induced subgraph on ancestors of the reward, along with a necessary and sufficient subset of latent confounders, to construct the set of possibly optimal arms. The regret incurred during this phase scales polynomially with respect to the number of nodes in the causal graph. The second phase involves the application of a standard bandit algorithm, such as the UCB algorithm. We also establish a regret bound for our two-phase approach, which is sublinear in the number of rounds.

Partial Structure Discovery is Sufficient for No-regret Learning in Causal Bandits

TL;DR

This work identifies that discovering the full causal structure is unnecessary; however, no existing work provides the necessary and sufficient components of the causal graph, and formally characterize the set of necessary and sufficient latent confounders one needs to detect or learn to ensure that all possibly optimal arms are identified correctly.

Abstract

Causal knowledge about the relationships among decision variables and a reward variable in a bandit setting can accelerate the learning of an optimal decision. Current works often assume the causal graph is known, which may not always be available a priori. Motivated by this challenge, we focus on the causal bandit problem in scenarios where the underlying causal graph is unknown and may include latent confounders. While intervention on the parents of the reward node is optimal in the absence of latent confounders, this is not necessarily the case in general. Instead, one must consider a set of possibly optimal arms/interventions, each being a special subset of the ancestors of the reward node, making causal discovery beyond the parents of the reward node essential. For regret minimization, we identify that discovering the full causal structure is unnecessary; however, no existing work provides the necessary and sufficient components of the causal graph. We formally characterize the set of necessary and sufficient latent confounders one needs to detect or learn to ensure that all possibly optimal arms are identified correctly. We also propose a randomized algorithm for learning the causal graph with a limited number of samples, providing a sample complexity guarantee for any desired confidence level. In the causal bandit setup, we propose a two-stage approach. In the first stage, we learn the induced subgraph on ancestors of the reward, along with a necessary and sufficient subset of latent confounders, to construct the set of possibly optimal arms. The regret incurred during this phase scales polynomially with respect to the number of nodes in the causal graph. The second phase involves the application of a standard bandit algorithm, such as the UCB algorithm. We also establish a regret bound for our two-phase approach, which is sublinear in the number of rounds.

Paper Structure

This paper contains 24 sections, 15 theorems, 60 equations, 4 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Setup
  3. Possibly Optimal Arms in Causal Bandits with Unknown Causal Graph
  4. Finite Sample Causal Discovery Algorithm
  5. Learning the Observable Graph
  6. Learning the Latent Confounders
  7. Algorithm for Causal Bandits with Unknown Graph Structure
  8. Experiments
  9. Conclusion
  10. Acknowledgment
  11. Supplemental Material
  12. Review of $d$-separation:
  13. Pearl's Rules of do-Calculus (pearl2009causality):
  14. Function to Detect Presence of Latent Confounder:
  15. Proof of Lemma \ref{['lemm-pomis_1']}:
  16. ...and 9 more sections

Key Result

Lemma 1

lee2018structural For causal graph $\mathcal{G}$ with reward $Y$, $\text{IB}(\mathcal{G}_{\overline{\textbf{W}}},Y)$is a POMIS, for any $\textbf{W} \subseteq \textbf{V} \setminus \{Y\}$.

Figures (4)

  • Figure 1: True Causal Graph $\mathcal{G}$ with four other graphs each with one missing bi-directed edge.
  • Figure 2: Simulations to demonstrate the advantage of Algorithm \ref{['sketch_band']} over full graph discovery (Learning all possible latents)
  • Figure 3: Simulations to demonstrate advantage of discovery for causal bandits.
  • Figure 4: Cumulative regret for Algorithm \ref{['sketch_band']} versus learning all possible latents $(\rho = \rho_L = 0.3)$.

Theorems & Definitions (17)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Theorem 2
  • ...and 7 more