DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects

TL;DR

DOLCE introduces a lagged-current decomposition to off-policy evaluation and learning, enabling unbiased estimation and optimization even when common support is violated. By weighting current actions with past-context-conditioned probabilities and decomposing rewards into lagged and current components, DOLCE achieves unbiasedness under local correctness and conditional independence and shows reduced bias-driven MSE in synthetic experiments. The method employs a two-step regression strategy to estimate the reward function components, improving both estimation accuracy and gradient-based policy optimization. Empirically, DOLCE consistently outperforms standard baselines (DM, IPS, DR) across varying degrees of positivity violations, data sizes, and action spaces, highlighting its practical relevance for real-world systems with non-overlapping support. The work suggests promising directions for extending DOLCE to fully deterministic policies and linking the approach to core causal-inference assumptions such as consistency, exchangeability, and positivity, to broaden its applicability.

Abstract

Off-policy evaluation (OPE) and off-policy learning (OPL) for contextual bandit policies leverage historical data to evaluate and optimize a target policy. Most existing OPE/OPL methods--based on importance weighting or imputation--assume common support between the target and logging policies. When this assumption is violated, these methods typically require unstable extrapolation, truncation, or conservative strategies for individuals outside the common support assumption. However, such approaches can be inadequate in settings where explicit evaluation or optimization for such individuals is required. To address this issue, we propose DOLCE: Decomposing Off-policy evaluation/learning into Lagged and Current Effects, a novel estimator that leverages contextual information from multiple time points to decompose rewards into lagged and current effects. By incorporating both past and present contexts, DOLCE effectively handles individuals who violate the common support assumption. We show that the proposed estimator is unbiased under two assumptions--local correctness and conditional independence. Our experiments demonstrate that DOLCE achieves substantial improvements in OPE and OPL, particularly as the proportion of individuals outside the common support assumption increases.

Figures (8)

• Figure 1: Comparison of the estimators' MSE with (i) varying the proportion of individuals who violate the common support assumption, (ii) varying logged data sizes, (iii) varying numbers of actions in the synthetic experiment.
• Figure 2: Comparison of the estimators' MSE with varying the scaling parameter $\lambda$ which controls the balance between lagged and current effects in the expected reward. The closer $\lambda$ is to $1$, the larger the impact of the current effect; the closer $\lambda$ is to $0$, the larger impact of the lagged effect.
• Figure 3: Comparison the relative policy value (reference: $V(\pi_0)$) of the OPL methods, with (i) varying the proportion of individuals who violate the full support assumption, (ii) varying training data sizes, (iii) varying numbers of actions in the synthetic experiment.
• Figure 4: Comparison the relative policy value (reference: $V(\pi_0)$) of the OPL methods, with varying the scaling parameter $\lambda$ which controls the balance between lagged and current effects in the expected reward. The closer $\lambda$ is to $1$, the larger the impact of the current effect; the closer $\lambda$ is to $0$, the larger impact of the lagged effect.
• Figure 5: Comparison of the estimators' statistical properties with varying the proportion of individuals who violate the common support assumption.

Theorems & Definitions (14)

• Proposition 2.2
• Proposition 2.3
• Proposition 3.4
• Proposition 3.5
• Proposition 3.6
• Proposition 3.8
• Proposition 3.9
• proof
• proof
• proof