Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Clustered KL-barycenter design for policy evaluation

Simon Weissmann, Till Freihaut, Claire Vernade, Giorgia Ramponi, Leif Döring

TL;DR

The paper addresses sample-efficient evaluation of multiple target policies in stochastic bandits by designing behavior policies that minimize information-theoretic divergence from the targets. It introduces KL-barycenter policy evaluation (KL-PE), where the behavior policy is the KL-barycenter of the target set, enabling bounded importance weights and favorable regret bounds; it then extends to clustered KL-based policy evaluation (CKL-PE) by partitioning targets into clusters and assigning a KL-barycenter per cluster to improve scalability. The authors establish upper and lower bounds on sample complexity, show that similarity between targets and the barycenter improves bounds, and demonstrate that clustering reduces the maximal importance weights, leading to better regret performance in practice. Empirical results on synthetic data with structured target policies confirm that CKL-PE reduces regret and scales to large target sets, suggesting practical impact for efficient offline data collection and policy selection in bandit-like settings and motivating future extensions to contextual bandits and MDPs.

Abstract

In the context of stochastic bandit models, this article examines how to design sample-efficient behavior policies for the importance sampling evaluation of multiple target policies. From importance sampling theory, it is well established that sample efficiency is highly sensitive to the KL divergence between the target and importance sampling distributions. We first analyze a single behavior policy defined as the KL-barycenter of the target policies. Then, we refine this approach by clustering the target policies into groups with small KL divergences and assigning each cluster its own KL-barycenter as a behavior policy. This clustered KL-based policy evaluation (CKL-PE) algorithm provides a novel perspective on optimal policy selection. We prove upper bounds on the sample complexity of our method and demonstrate its effectiveness with numerical validation.

Clustered KL-barycenter design for policy evaluation

TL;DR

The paper addresses sample-efficient evaluation of multiple target policies in stochastic bandits by designing behavior policies that minimize information-theoretic divergence from the targets. It introduces KL-barycenter policy evaluation (KL-PE), where the behavior policy is the KL-barycenter of the target set, enabling bounded importance weights and favorable regret bounds; it then extends to clustered KL-based policy evaluation (CKL-PE) by partitioning targets into clusters and assigning a KL-barycenter per cluster to improve scalability. The authors establish upper and lower bounds on sample complexity, show that similarity between targets and the barycenter improves bounds, and demonstrate that clustering reduces the maximal importance weights, leading to better regret performance in practice. Empirical results on synthetic data with structured target policies confirm that CKL-PE reduces regret and scales to large target sets, suggesting practical impact for efficient offline data collection and policy selection in bandit-like settings and motivating future extensions to contextual bandits and MDPs.

Abstract

In the context of stochastic bandit models, this article examines how to design sample-efficient behavior policies for the importance sampling evaluation of multiple target policies. From importance sampling theory, it is well established that sample efficiency is highly sensitive to the KL divergence between the target and importance sampling distributions. We first analyze a single behavior policy defined as the KL-barycenter of the target policies. Then, we refine this approach by clustering the target policies into groups with small KL divergences and assigning each cluster its own KL-barycenter as a behavior policy. This clustered KL-based policy evaluation (CKL-PE) algorithm provides a novel perspective on optimal policy selection. We prove upper bounds on the sample complexity of our method and demonstrate its effectiveness with numerical validation.

Paper Structure

This paper contains 34 sections, 14 theorems, 85 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Outline.
  3. Formalization of the problem and approach
  4. Best-policy selection.
  5. Behavior policy design.
  6. Policy Evaluation via KL-barycenter.
  7. Clustered KL-barycenter design.
  8. Related Work
  9. Behavior policy optimization.
  10. Off-policy evaluation.
  11. Structured Policy Evaluation with KL-Barycenter
  12. Regret Upper Bound for KL-Barycenters
  13. Lower bound
  14. On the Relationship Between the KL-Barycenter and the Target Policies
  15. Best policy selection using clustered policy evaluation
  16. ...and 19 more sections

Key Result

Lemma 2.1

Let $\Pi_N = \{\pi_1, \ldots, \pi_N\}$ be a set of strictly positive sum-normalized target policies. Then the arithmetic mean $\pi_{\mathrm{KL}}:= \frac{1}{N} \sum_{i=1}^N \pi_i$ solves eq:Opti_KLbarycenter. We call this policy the KL-barycenter.

Figures (1)

  • Figure 1: Comparison of (a) the values of $M\sigma^2_c$ and (b) the regret for different numbers of clusters $M$ averaged over $1000$ independent runs.

Theorems & Definitions (29)

  • Lemma 2.1
  • Lemma 4.1
  • proof
  • Proposition 4.2
  • Example 4.3
  • Proposition 4.4
  • Remark 4.5
  • Proposition 4.6
  • Proposition 5.2
  • Remark 5.3
  • ...and 19 more