Counterfactual Learning of Stochastic Policies with Continuous Actions

TL;DR

This paper tackles offline learning of stochastic policies with continuous actions under counterfactual risk minimization (CRM). It introduces a joint kernel embedding to model actions and contexts, leading to the Counterfactual Loss Predictor (CLP) that enables differentiable, expressive policy learning over continuous actions via a soft-argmin over anchor points. To address optimization and estimation challenges, the authors propose a differentiable soft-clipping estimator (scIPS) and employ proximal point algorithms to improve non-convex CRM objectives, along with theoretical excess-risk guarantees. They also establish an offline evaluation protocol and release CoCoA, a large-scale real-world logged dataset for continuous-action CRM, accompanied by extensive empirical validation across synthetic, semi-synthetic, and real data showing that CLP with scIPS and PPA outperforms discretization and several baselines. The work advances practical CRM for continuous actions, enabling reliable offline evaluation, improved optimization, and scalable kernel-based modeling for real-world decision tasks such as web advertising and healthcare dosing.

Abstract

Counterfactual reasoning from logged data has become increasingly important for many applications such as web advertising or healthcare. In this paper, we address the problem of learning stochastic policies with continuous actions from the viewpoint of counterfactual risk minimization (CRM). While the CRM framework is appealing and well studied for discrete actions, the continuous action case raises new challenges about modelization, optimization, and~offline model selection with real data which turns out to be particularly challenging. Our paper contributes to these three aspects of the CRM estimation pipeline. First, we introduce a modelling strategy based on a joint kernel embedding of contexts and actions, which overcomes the shortcomings of previous discretization approaches. Second, we empirically show that the optimization aspect of counterfactual learning is important, and we demonstrate the benefits of proximal point algorithms and smooth estimators. Finally, we propose an evaluation protocol for offline policies in real-world logged systems, which is challenging since policies cannot be replayed on test data, and we release a new large-scale dataset along with multiple synthetic, yet realistic, evaluation setups.

Figures (20)

• Figure 1: Illustration of the joint kernel embedding for the counterfactual loss predictor (CLP) and loss estimator.
• Figure 2: Soft Clipping of importance sampling weight $w$.
• Figure 3: Causal Graph of the synthetic setting. $A$ denotes action, $X$ context, $G$ unobserved group label, $Y$ outcome and $P$ unobserved potentials. Unobserved elements are dotted.
• Figure 4: Continuous vs discretization strategies. Test rewards on NoisyMoons dataset with varying numbers of anchor points for our continuous parametrization for IPS and SDM, versus naive discretization with softmax policies. Note that few anchor points are sufficient to achieve good results on this dataset; this is not the case for more complicated ones (e.g., Warfarin requires at least 15 anchor points).
• Figure 5: Influence of soft-clipping. Relative improvements in the test performance for soft- vs hard-clipping on synthetic datasets. The points correspond to different choices of the clipping parameter, models and initialization.

Theorems & Definitions (10)

• Proposition 4.1: Generalization bound for $\hat{L}_\text{scIPS}(\pi)$
• Remark 4.1
• Remark 4.2
• Proposition 5.1: Excess risk upper bound
• Example D.1
• Definition E.1: epsilon covering and metric entropy
• proof
• Proposition E.1
• proof
• proof