CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

TL;DR

This work tackles safe off-policy evaluation for contextual bandits by integrating counterfactual annotations into a doubly robust framework. It introduces three DR-inspired estimators—DM+-IS, DM-IS+, and DM+-IS+—and analyzes their bias and variance under imperfect annotations and reward-model misspecification. Theoretical results show that, in realistic settings with biased or noisy annotations, leveraging annotations in the reward-model (DM) part yields the strongest robustness, with DM+-IS often outperforming alternatives in practice. Empirical results across multiple contextual-bandit environments validate these insights and yield practical guidance for estimator selection in high-stakes deployment scenarios.

Abstract

Off-policy evaluation (OPE) provides safety guarantees by estimating the performance of a policy before deployment. Recent work introduced IS+, an importance sampling (IS) estimator that uses expert-annotated counterfactual samples to improve behavior dataset coverage. However, IS estimators are known to have high variance; furthermore, the performance of IS+ deteriorates when annotations are imperfect. In this work, we propose a family of OPE estimators inspired by the doubly robust (DR) principle. A DR estimator combines IS with a reward model estimate, known as the direct method (DM), and offers favorable statistical guarantees. We propose three strategies for incorporating counterfactual annotations into a DR-inspired estimator and analyze their properties under various realistic settings. We prove that using imperfect annotations in the DM part of the estimator best leverages the annotations, as opposed to using them in the IS part. To support our theoretical findings, we evaluate the proposed estimators in three contextual bandit environments. Our empirical results show that when the reward model is misspecified and the annotations are imperfect, it is most beneficial to use the annotations only in the DM portion of a DR estimator. Based on these theoretical and empirical insights, we provide a practical guide for using counterfactual annotations in different realistic settings.

Figures (8)

• Figure 1: Counterfactual annotated dataset example with two contexts and two actions: We have two factual samples $(s_1, a_1, r_1), (s_2, a_1, r_2)$. For the first (left) factual sample, we have a corresponding counterfactual annotation $(s_1, a_2, g_1^{a_2})$. For the second (right) factual sample, the counterfactual annotation is missing.
• Figure 2: Heatmap of mean RMSE with a well-specified reward model in the 2-context bandit setting (lower RMSE is represented lighter): The bias of counterfactual annotations has a larger impact on RMSE than the variance. The $x,y$-axis represents the variance ($\Delta_G$) and the bias ($\epsilon_G$) of the annotations, respectively. The RMSE hardly varies across the x-axis but increases proportionally to the magnitude of the annotation bias. This trend is particularly salient in $\textup{DM-IS+}$ and $\textup{DM+-IS+}$. $\textup{DM+-IS}$ is most robust to imperfect annotations because the RMSE is more consistent regardless of the annotation bias/variance.
• Figure 3: Heatmaps of mean RMSE with a misspecified reward model and imperfect annotations.(lower RMSE is represented lighter). The $x$-axis represents annotation bias, $\epsilon_G$. Across all datasets, $\textup{DM+-IS}$ performs either better than all baselines, or comparably to the best-performing baseline. Among all methods that use counterfactual annotations, $\textup{DM+-IS}$ is most robust to biased annotations and a misspecified reward model. In comparison to baselines that do not use counterfactual annotations, $\textup{DM+-IS}$ frequently produces a lower RMSE.
• Figure 4: Lookup table capturing the practical considerations when choosing an OPE estimator. The most critical factors include whether the reward model is misspecified or not and the quality of the annotations. If the reward model and annotation equality is known a priori, the best OPE estimator can be easily identified.
• Figure 5: Exploring the consequences of always using $\textup{DM+-IS}$ in the sepsis environment. In the vast majority of cases, the reward model or annotation quality is unknown. Regardless, we choose $\textup{DM+-IS}$ to learn an OPE estimate and compare it to the best performing OPE method. We find that regardless of how imperfect the annotations are, $\textup{DM+-IS}$ produces estimates with low mean $\Delta$ relative to the range of reward in the Sepsis environment, which is 5.2. The color of the dots represents the magnitude of mean $\Delta$, and the size is proportional to the variance of $\Delta$. $\Delta$ increases in magnitude with the increase in annotation bias ($\epsilon_G$), though even in the most extreme cases, $\Delta$ is relatively small.

Theorems & Definitions (23)

• Proposition 1: name=Unbiasedness of $\textup{DM+-IS}$ under imperfect annotations
• Theorem 2: name=Bias of $\textup{DM-IS+}$ and $\textup{DM+-IS+}$ under imperfect annotations
• Theorem 3: name=Variance of $\textup{DM+-IS}$ under imperfect annotations
• Proposition 4: name=Unbiasedness of weighted reward function
• Proposition 5
• Proposition 6
• Proposition 7
• Proposition 8
• Proposition 9
• Proposition 10: Unbiasedness of $\hat{V}^{\text{DM}}$