Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Policy Learning with Weight Clipping and Heaviside Composite Optimization

Jingren Liu, Hanzhang Qin, Junyi Liu, Mabel C. Chou, Jong-Shi Pang

TL;DR

The paper tackles offline policy learning under weak overlap, where standard IPW/DR estimators exhibit high variance. It introduces the Optimized Clipped Doubly Robust (OCDR) estimator with a closed-form, policy-dependent clipping threshold $\widehat{\tau}(g)$ that minimizes the policy-value MSE, and recasts policy optimization as a Heaviside composite optimization (HSCOP). A Progressive Integer Programming (PIP) method is developed to efficiently solve the resulting single-level HSCOP reformulation, enabling tractable learning in a linear policy class. The authors prove a suboptimality bound showing that reducing the worst-case MSE in policy evaluation improves learning performance, and validate the approach with synthetic and real-world insurance data, demonstrating computational gains and superior policy value. Overall, OCDRL offers a principled, scalable framework for variance-reduced offline policy learning with theoretical guarantees and practical impact.

Abstract

Offline policy learning aims to use historical data to learn an optimal personalized decision rule. In the standard estimate-then-optimize framework, reweighting-based methods (e.g., inverse propensity weighting or doubly robust estimators) are widely used to produce unbiased estimates of policy values. However, when the propensity scores of some treatments are small, these reweighting-based methods suffer from high variance in policy value estimation, which may mislead the downstream policy optimization and yield a learned policy with inferior value. In this paper, we systematically develop an offline policy learning algorithm based on a weight-clipping estimator that truncates small propensity scores via a clipping threshold chosen to minimize the mean squared error (MSE) in policy value estimation. Focusing on linear policies, we address the bilevel and discontinuous objective induced by weight-clipping-based policy optimization by reformulating the problem as a Heaviside composite optimization problem, which provides a rigorous computational framework. The reformulated policy optimization problem is then solved efficiently using the progressive integer programming method, making practical policy learning tractable. We establish an upper bound for the suboptimality of the proposed algorithm, which reveals how the reduction in MSE of policy value estimation, enabled by our proposed weight-clipping estimator, leads to improved policy learning performance.

Offline Policy Learning with Weight Clipping and Heaviside Composite Optimization

TL;DR

The paper tackles offline policy learning under weak overlap, where standard IPW/DR estimators exhibit high variance. It introduces the Optimized Clipped Doubly Robust (OCDR) estimator with a closed-form, policy-dependent clipping threshold that minimizes the policy-value MSE, and recasts policy optimization as a Heaviside composite optimization (HSCOP). A Progressive Integer Programming (PIP) method is developed to efficiently solve the resulting single-level HSCOP reformulation, enabling tractable learning in a linear policy class. The authors prove a suboptimality bound showing that reducing the worst-case MSE in policy evaluation improves learning performance, and validate the approach with synthetic and real-world insurance data, demonstrating computational gains and superior policy value. Overall, OCDRL offers a principled, scalable framework for variance-reduced offline policy learning with theoretical guarantees and practical impact.

Abstract

Offline policy learning aims to use historical data to learn an optimal personalized decision rule. In the standard estimate-then-optimize framework, reweighting-based methods (e.g., inverse propensity weighting or doubly robust estimators) are widely used to produce unbiased estimates of policy values. However, when the propensity scores of some treatments are small, these reweighting-based methods suffer from high variance in policy value estimation, which may mislead the downstream policy optimization and yield a learned policy with inferior value. In this paper, we systematically develop an offline policy learning algorithm based on a weight-clipping estimator that truncates small propensity scores via a clipping threshold chosen to minimize the mean squared error (MSE) in policy value estimation. Focusing on linear policies, we address the bilevel and discontinuous objective induced by weight-clipping-based policy optimization by reformulating the problem as a Heaviside composite optimization problem, which provides a rigorous computational framework. The reformulated policy optimization problem is then solved efficiently using the progressive integer programming method, making practical policy learning tractable. We establish an upper bound for the suboptimality of the proposed algorithm, which reveals how the reduction in MSE of policy value estimation, enabled by our proposed weight-clipping estimator, leads to improved policy learning performance.
Paper Structure (51 sections, 16 theorems, 132 equations, 4 figures, 2 tables, 2 algorithms)

This paper contains 51 sections, 16 theorems, 132 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Main Results and Contributions
  3. Related Work
  4. Preliminaries on Offline Policy Learning
  5. Offline Data Description
  6. Policy Learning and Performance Metric
  7. Policy Evaluation Using Reweighting-Based Methods
  8. Weight-Clipping-Based Policy Learning: Formulation and Challenges
  9. (Optimized) Clipped Doubly Robust Estimator
  10. Challenges in OCDR-Based Policy Optimization
  11. Mathematical Reformulations for Policy Optimization
  12. Explicit Solution to the MSE Minimization Problem (\ref{['program: optimal tau']})
  13. HSCOP Formulation of the Policy Optimization Problem (\ref{['program:policy optimization']})
  14. Algorithms for Policy Optimization
  15. Computation via Mixed Integer Programming
  16. ...and 36 more sections

Key Result

Proposition 1

Denote $\widehat{\Gamma}^{\mathrm{DR}}_s(g)$ by the doubly robust score in the DR estimator. Under Assumption asp:dgp, suppose $L' \leq \mathrm{Var}(Y_s \mid X^s) \leq U'$ for all $s \in [N]$ and $U'\geq L' \geq 0$. For any policy $g \in \mathcal{G}_\mathrm{LPC}$, there exist positive constants $U

Figures (4)

  • Figure 1: A Toy Example
  • Figure 2: Out-of-Sample Suboptimality Gaps
  • Figure 3: Treatment Selection Behavior of Different Algorithms
  • Figure EC.1: Targeting Policy Learned by OCDRL

Theorems & Definitions (22)

  • Proposition 1
  • Definition 1: Clipped Doubly Robust Estimator
  • Proposition 2
  • Definition 2: Optimized Clipped Doubly Robust Estimator
  • Definition 3: Heaviside Composite Optimization Problem
  • Proposition 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • Theorem 1
  • ...and 12 more