Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendation
Siyu Wang, Xiaocong Chen, Lina Yao
TL;DR
PGCR addresses the challenge of high-dimensional, noisy, and non-causally relevant state information in offline RL-based recommender systems by explicitly isolating causally relevant components (CRCs) from causally irrelevant components (CIRCs) in the state. It proposes a two-stage framework: first, a causal feature selection policy that intervenes on actions and uses a Wasserstein distance-based reward $W_1$ to preserve CRCs, and second, an encoder trained with an MSE objective to map states to CRC-focused latent representations. The authors prove identifiability of causal effects under interventions and show empirically that CRC-focused representations improve both offline policy performance and online CTR in simulation. This approach enables offline RLRS to learn more robust decision-making with limited data, offering practical benefits for scalable, privacy-preserving recommender systems.
Abstract
In offline reinforcement learning-based recommender systems (RLRS), learning effective state representations is crucial for capturing user preferences that directly impact long-term rewards. However, raw state representations often contain high-dimensional, noisy information and components that are not causally relevant to the reward. Additionally, missing transitions in offline data make it challenging to accurately identify features that are most relevant to user satisfaction. To address these challenges, we propose Policy-Guided Causal Representation (PGCR), a novel two-stage framework for causal feature selection and state representation learning in offline RLRS. In the first stage, we learn a causal feature selection policy that generates modified states by isolating and retaining only the causally relevant components (CRCs) while altering irrelevant components. This policy is guided by a reward function based on the Wasserstein distance, which measures the causal effect of state components on the reward and encourages the preservation of CRCs that directly influence user interests. In the second stage, we train an encoder to learn compact state representations by minimizing the mean squared error (MSE) loss between the latent representations of the original and modified states, ensuring that the representations focus on CRCs. We provide a theoretical analysis proving the identifiability of causal effects from interventions, validating the ability of PGCR to isolate critical state components for decision-making. Extensive experiments demonstrate that PGCR significantly improves recommendation performance, confirming its effectiveness for offline RL-based recommender systems.