Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Offline Policy Learning with Observational Data from Multiple Sources

Aldo Gael Carranza, Susan Athey

TL;DR

The problem of using observational bandit feedback data from multiple heterogeneous data sources to learn a personalized decision policy that robustly generalizes across diverse target settings is considered, and a policy learning algorithm tailored to this objective is developed.

Abstract

We consider the problem of using observational bandit feedback data from multiple heterogeneous data sources to learn a personalized decision policy that robustly generalizes across diverse target settings. To achieve this, we propose a minimax regret optimization objective to ensure uniformly low regret under general mixtures of the source distributions. We develop a policy learning algorithm tailored to this objective, combining doubly robust offline policy evaluation techniques and no-regret learning algorithms for minimax optimization. Our regret analysis shows that this approach achieves the minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources. Our analysis, extensions, and experimental results demonstrate the benefits of this approach for learning robust decision policies from multiple data sources.

Robust Offline Policy Learning with Observational Data from Multiple Sources

TL;DR

The problem of using observational bandit feedback data from multiple heterogeneous data sources to learn a personalized decision policy that robustly generalizes across diverse target settings is considered, and a policy learning algorithm tailored to this objective is developed.

Abstract

We consider the problem of using observational bandit feedback data from multiple heterogeneous data sources to learn a personalized decision policy that robustly generalizes across diverse target settings. To achieve this, we propose a minimax regret optimization objective to ensure uniformly low regret under general mixtures of the source distributions. We develop a policy learning algorithm tailored to this objective, combining doubly robust offline policy evaluation techniques and no-regret learning algorithms for minimax optimization. Our regret analysis shows that this approach achieves the minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources. Our analysis, extensions, and experimental results demonstrate the benefits of this approach for learning robust decision policies from multiple data sources.

Paper Structure

This paper contains 64 sections, 19 theorems, 166 equations, 3 figures, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. PRELIMINARIES
  4. Setting
  5. Objective
  6. Data-Generating Processes
  7. Data Assumptions
  8. APPROACH
  9. Nuisance Parameters
  10. Policy Value and Regret Estimators
  11. Optimization Objective
  12. ALGORITHM
  13. OPO Oracle
  14. Exponentiated Gradient OPO
  15. REGRET BOUNDS
  16. ...and 49 more sections

Key Result

Theorem 1

Suppose Assumptions ass:dgp, ass:LocalDataSizeScaling, and ass:FiniteSampleError hold. For any $\epsilon>0$, let $\Lambda_\epsilon$ denote a minimal $\epsilon$-covering set of $\Lambda$ under the $\ell_1$ distance. Set $T=(n/\mathfrak{s}(\Lambda\|\bar{n}))^{1+\alpha}$ for any choice of $\alpha>0$. T for any ${\lambda'}\in\Lambda$, where where the constant $B=\max_{s\in\mathcal{S}} B_s$ is a unifo

Figures (3)

  • Figure 1: Empirical source regret $R_1(\pi)$.
  • Figure 2: Empirical mixture regret $R_\lambda(\pi)$.
  • Figure : EG-OPO

Theorems & Definitions (43)

  • Definition 1: Mixture Policy Value
  • Definition 2: Mixture Regret
  • Remark
  • Definition 3: Nuisance Parameters
  • Definition 4: OPO Oracle
  • Definition 5: Entropy integral
  • Definition 6: Skewness
  • Theorem 1: Mixture-Agnostic Regret Bound
  • Lemma 1
  • Theorem 2: Target Regret Bound
  • ...and 33 more