Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foresighted Online Policy Optimization with Interference

Liner Xiang, Jiayi Wang, Hengrui Cai

TL;DR

This work addresses online contextual bandits under interference, where past actions influence future rewards through a known interference structure. It introduces FRONT, a foresighted online policy that integrates an online additive outcome model with heterogeneous treatment effects and homogeneous interference, plus an exposure-mapped, low-dimensional representation and an augmented exploration strategy with force pulls. The authors establish tail bounds, consistency, and asymptotic normality of the online estimator, derive conditions for interference convergence, and prove sublinear regrets under two definitions and inference regimes. Empirical validation includes comprehensive simulations and a real-world hotel-pricing case, demonstrating FRONT's superior long-term performance and valid online statistical inference. The framework advances online decision-making in networks with interference and provides practical guidance for deployment and future extensions to broader models and reinforcement-learning settings.

Abstract

Contextual bandits, which leverage the baseline features of sequentially arriving individuals to optimize cumulative rewards while balancing exploration and exploitation, are critical for online decision-making. Existing approaches typically assume no interference, where each individual's action affects only their own reward. Yet, such an assumption can be violated in many practical scenarios, and the oversight of interference can lead to short-sighted policies that focus solely on maximizing the immediate outcomes for individuals, which further results in suboptimal decisions and potentially increased regret over time. To address this significant gap, we introduce the foresighted online policy with interference (FRONT) that innovatively considers the long-term impact of the current decision on subsequent decisions and rewards. The proposed FRONT method employs a sequence of exploratory and exploitative strategies to manage the intricacies of interference, ensuring robust parameter inference and regret minimization. Theoretically, we establish a tail bound for the online estimator and derive the asymptotic distribution of the parameters of interest under suitable conditions on the interference network. We further show that FRONT attains sublinear regret under two distinct definitions, capturing both the immediate and consequential impacts of decisions, and we establish these results with and without statistical inference. The effectiveness of FRONT is further demonstrated through extensive simulations and a real-world application to urban hotel profits.

Foresighted Online Policy Optimization with Interference

TL;DR

This work addresses online contextual bandits under interference, where past actions influence future rewards through a known interference structure. It introduces FRONT, a foresighted online policy that integrates an online additive outcome model with heterogeneous treatment effects and homogeneous interference, plus an exposure-mapped, low-dimensional representation and an augmented exploration strategy with force pulls. The authors establish tail bounds, consistency, and asymptotic normality of the online estimator, derive conditions for interference convergence, and prove sublinear regrets under two definitions and inference regimes. Empirical validation includes comprehensive simulations and a real-world hotel-pricing case, demonstrating FRONT's superior long-term performance and valid online statistical inference. The framework advances online decision-making in networks with interference and provides practical guidance for deployment and future extensions to broader models and reinforcement-learning settings.

Abstract

Contextual bandits, which leverage the baseline features of sequentially arriving individuals to optimize cumulative rewards while balancing exploration and exploitation, are critical for online decision-making. Existing approaches typically assume no interference, where each individual's action affects only their own reward. Yet, such an assumption can be violated in many practical scenarios, and the oversight of interference can lead to short-sighted policies that focus solely on maximizing the immediate outcomes for individuals, which further results in suboptimal decisions and potentially increased regret over time. To address this significant gap, we introduce the foresighted online policy with interference (FRONT) that innovatively considers the long-term impact of the current decision on subsequent decisions and rewards. The proposed FRONT method employs a sequence of exploratory and exploitative strategies to manage the intricacies of interference, ensuring robust parameter inference and regret minimization. Theoretically, we establish a tail bound for the online estimator and derive the asymptotic distribution of the parameters of interest under suitable conditions on the interference network. We further show that FRONT attains sublinear regret under two distinct definitions, capturing both the immediate and consequential impacts of decisions, and we establish these results with and without statistical inference. The effectiveness of FRONT is further demonstrated through extensive simulations and a real-world application to urban hotel profits.
Paper Structure (46 sections, 13 theorems, 147 equations, 12 figures, 1 algorithm)

This paper contains 46 sections, 13 theorems, 147 equations, 12 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Framework
  4. Intuitive Policies and Working Model
  5. The Proposed Method: FRONT
  6. Theory
  7. Consistency of Estimated Parameters
  8. Statistical Inference
  9. Convergence of $\kappa_t$
  10. Asymptotic Normality
  11. Regret Analysis
  12. With Inference
  13. Without Inference
  14. Simulation Studies
  15. Simulation Setup
  16. ...and 31 more sections

Key Result

Proposition 1

Under the conditional mean outcome model in Equation equ:conditionalmean, the optimal policy $\pi^\ast(\cdot)$ at time $t$ is given by

Figures (12)

  • Figure 1: The causal graphs illustrate interference in online decision making. Individuals arrive sequentially with triples ${\bm{x}_t, a_t, y_t}$, where $\bm{x}_t$ are context features, $a_t$ is the chosen action, and $y_t$ the outcome. Upper plot (a): Green arrows show how past actions influence the outcome of the $t$-th individual. Lower plot (b): Purple arrows show how the $t$-th individual affects subsequent outcomes. Line width reflects weight magnitude.
  • Figure 2: ISO over time. Each small rectangle represents the ISO generated by an individual’s action. Yellow rectangles indicate interference realized up to time $T$, while blue rectangles represent ongoing interference extending beyond $T$. By $T^\prime$, all interference caused before $T$ has manifested. Note that $T^\prime$ may be infinite.
  • Figure 3: Upper panel: the cumulative average reward over time with growing interference scale $g(t)$. Lower panel: the cumulative average regret, quantifying the gap between the optimal policy and the other three policies (shown in the upper panel), with regret defined by $R_1(T)$ with Equation \ref{['equ:regret1']}.
  • Figure 4: Consistency and asymptotic normality of the online estimator when $g(t) = \lfloor 0.2t \rfloor$ is shown in the figure. Each ray represents a parameter, with dot colors transitioning from light to dark as the time steps increase.Left panel: Average SE/MCSD, with the red circle indicating the nominal level of 1. Middle panel: Average bias, with the red circle indicating the nominal level of 0. Right panel: Coverage probability, with the red circle indicating the nominal level of 95%.
  • Figure 5: Consistency and asymptotic normality of the online estimator when $g(t)= \lfloor 5\sqrt{t} \rfloor$.
  • ...and 7 more figures

Theorems & Definitions (21)

  • Proposition 1: Optimal policy with interference
  • Theorem 2: Tail bound for the online estimator
  • Corollary 3: Consistency of the online estimator
  • Theorem 4: Convergence of the Optimal Interference Action
  • Theorem 5: Convergence of $\kappa_t$
  • Corollary 6: Growing interference scale with equal weights
  • Theorem 7: Asymptotic normality for the online estimator
  • Theorem 8: Regret bound
  • Theorem 9: Regret bound without inference
  • proof
  • ...and 11 more