Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Bandits

TL;DR

The paper tackles off-policy evaluation and learning in contextual combinatorial bandits, where policies select action subsets and the subset space grows exponentially. It introduces a factored action space and a reward decomposition into a main effect and a residual, yielding the OPCB estimator that applies importance sampling to the main actions while using regression for the residual, achieving substantial variance reduction and potential bias control. Theoretical results guarantee unbiasedness under mild conditions and characterize variance, and the method includes data-driven procedures to identify main actions and an extension to off-policy learning via policy gradients (OPCB-PG). Empirical results on synthetic and real-world data (KuaiRec) show OPCB outperforms standard baselines in both OPE and OPL, especially in large or data-scarce settings, highlighting its practical impact for recommender systems and healthcare applications. Overall, OPCB provides a principled, scalable approach to safe and efficient offline evaluation and optimization of combinatorial policies.

Abstract

We explore off-policy evaluation and learning (OPE/L) in contextual combinatorial bandits (CCB), where a policy selects a subset in the action space. For example, it might choose a set of furniture pieces (a bed and a drawer) from available items (bed, drawer, chair, etc.) for interior design sales. This setting is widespread in fields such as recommender systems and healthcare, yet OPE/L of CCB remains unexplored in the relevant literature. Typical OPE/L methods such as regression and importance sampling can be applied to the CCB problem, however, they face significant challenges due to high bias or variance, exacerbated by the exponential growth in the number of available subsets. To address these challenges, we introduce a concept of factored action space, which allows us to decompose each subset into binary indicators. This formulation allows us to distinguish between the ''main effect'' derived from the main actions, and the ''residual effect'', originating from the supplemental actions, facilitating more effective OPE. Specifically, our estimator, called OPCB, leverages an importance sampling-based approach to unbiasedly estimate the main effect, while employing regression-based approach to deal with the residual effect with low variance. OPCB achieves substantial variance reduction compared to conventional importance sampling methods and bias reduction relative to regression methods under certain conditions, as illustrated in our theoretical analysis. Experiments demonstrate OPCB's superior performance over typical methods in both OPE and OPL.

Figures (12)

• Figure 1: The graphical model of reward function decomposition into the main and residual effects, where $\phi(m)$ is the set of main actions. In this example, we consider a total outfit coordination where we recommend a combination of the fashion items, and we regard the tops and bottoms as the main actions, whereas the residual effect captures the effect of the accessories on the reward.
• Figure 2: Comparison of the estimators' MSE, Squared Bias, and Variance with varying logged data sizes ($n$).
• Figure 5: Comparison of the estimators' MSE, Squared Bias, and Variance with varying numbers of main actions in OPCB ($\hat{\phi}$).
• Figure 6: Comparison of the estimators' MSE, Squared Bias, and Variance with varying logged data sizes ($n$) on KuaiRec.
• Figure 7: Comparison of the policy value (normalized by $V(\pi_0)$ ) of the OPL methods, with varying (i) training data sizes and (ii) the number of unique actions.

Theorems & Definitions (20)

• Example 1.1: Total Outfit Coordination
• Example 1.2: Precision Medicine
• Theorem 3.3: Unbiasedness of the OPCB estimator
• Theorem 3.4: Bias of OPCB
• Theorem 3.5: Variance of OPCB
• Theorem C.2: Unbiasedness of the gradient of the OPCB estimator
• Theorem C.3: Bias of the gradient of the OPCB estimator
• Theorem C.4: Variance of the gradient of the OPCB estimator
• proof
• proof