Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Policy Maximization Under Network Interference

Aidan Gleich, Eric Laber, Alexander Volfovsky

TL;DR

This work addresses scalable policy optimization under network interference in dynamic graphs. It introduces a SANIA-based linear reward formulation that converts node-level rewards into a linear model, enabling a Thompson sampling algorithm (networkUCL) to learn parameters while maximizing total reward. The authors prove a Bayesian regret bound of $O(D\sqrt{nT}\log(nT))$ and demonstrate rapid learning in large-scale simulations, outperforming existing network-bandit methods and maintaining robustness under misspecification. By bridging causal inference assumptions with scalable bandit algorithms, the approach makes policy optimization feasible for large, evolving networks and lays groundwork for extensions to partially observed networks and more complex interference patterns.

Abstract

Many interventions, such as vaccines in clinical trials or coupons in online marketplaces, must be assigned sequentially without full knowledge of their effects. Multi-armed bandit algorithms have proven successful in such settings. However, standard independence assumptions fail when the treatment status of one individual impacts the outcomes of others, a phenomenon known as interference. We study optimal-policy learning under interference on a dynamic network. Existing approaches to this problem require repeated observations of the same fixed network and struggle to scale in sample size beyond as few as fifteen connected units -- both limit applications. We show that under common assumptions on the structure of interference, rewards become linear. This enables us to develop a scalable Thompson sampling algorithm that maximizes policy impact when a new $n$-node network is observed each round. We prove a Bayesian regret bound that is sublinear in $n$ and the number of rounds. Simulation experiments show that our algorithm learns quickly and outperforms existing methods. The results close a key scalability gap between causal inference methods for interference and practical bandit algorithms, enabling policy optimization in large-scale networked systems.

Scalable Policy Maximization Under Network Interference

TL;DR

This work addresses scalable policy optimization under network interference in dynamic graphs. It introduces a SANIA-based linear reward formulation that converts node-level rewards into a linear model, enabling a Thompson sampling algorithm (networkUCL) to learn parameters while maximizing total reward. The authors prove a Bayesian regret bound of and demonstrate rapid learning in large-scale simulations, outperforming existing network-bandit methods and maintaining robustness under misspecification. By bridging causal inference assumptions with scalable bandit algorithms, the approach makes policy optimization feasible for large, evolving networks and lays groundwork for extensions to partially observed networks and more complex interference patterns.

Abstract

Many interventions, such as vaccines in clinical trials or coupons in online marketplaces, must be assigned sequentially without full knowledge of their effects. Multi-armed bandit algorithms have proven successful in such settings. However, standard independence assumptions fail when the treatment status of one individual impacts the outcomes of others, a phenomenon known as interference. We study optimal-policy learning under interference on a dynamic network. Existing approaches to this problem require repeated observations of the same fixed network and struggle to scale in sample size beyond as few as fifteen connected units -- both limit applications. We show that under common assumptions on the structure of interference, rewards become linear. This enables us to develop a scalable Thompson sampling algorithm that maximizes policy impact when a new -node network is observed each round. We prove a Bayesian regret bound that is sublinear in and the number of rounds. Simulation experiments show that our algorithm learns quickly and outperforms existing methods. The results close a key scalability gap between causal inference methods for interference and practical bandit algorithms, enabling policy optimization in large-scale networked systems.

Paper Structure

This paper contains 32 sections, 4 theorems, 25 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Interference and Peer Influence Effects
  4. Bandits with Interference
  5. Model Setup
  6. Problem Background
  7. Reward Functions
  8. Network Structure
  9. Latent Space Models
  10. Exponential Random Graphs (ERGM)
  11. Stochastic Block Model (SBM)
  12. Algorithm
  13. Linear Regression Formulation
  14. Design Matrix
  15. Prior Distribution
  16. ...and 17 more sections

Key Result

Theorem 1

Assume there exist positive constants $c_1$, $c_2$ such that $\sup_{\boldsymbol{\theta} \in \Theta} \| \boldsymbol{\theta}\|_2\leq c_1$ and $\sup_{\mathbf{X} \in \mathcal{X}^n} \|\mathbf{X}\|_2 \leq c_2$ for all $n$, and suppose Assumptions ass1--ass4 hold. Algorithm alg1 then satisfies

Figures (4)

  • Figure 1: A random sample from the population network is drawn at each time period.
  • Figure 2: In \ref{['fig1:sub_sparse_n']}, we simulate from a sparse SBM with expected degree of about $4$ and compare our method across moderate network sizes. We see that larger $n$ values accumulate regret faster initially but stabilize at around the same time as smaller values. In \ref{['fig1:agarwal_vs_sania']}, we compare our method to Algorithm 1 of agarwal2024 for networks of size $n=8$ with SANIA reward functions.
  • Figure 3: We investigate the impact of the prior mean and budget constraint on cumulative regret for the sparse SBM with $n=100$.
  • Figure 4: Comparison to agarwal2024 under failure of the SANIA assumptions.

Theorems & Definitions (7)

  • Theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 2
  • proof