Breaking the Curse of Repulsion: Optimistic Distributionally Robust Policy Optimization for Off-Policy Generative Recommendation

TL;DR

This work tackles the brittleness of offline policy-based reinforcement learning in generative recommendation under heavy-tailed data. It introduces the Divergence Theory of Repulsive Optimization, showing that negative gradient updates cause exponential parameter-space expansion and model collapse. To address this, it formulates an Optimistic Distributionally Robust Optimization objective whose closed-form solution enforces hard filtering of noise via top-$ppa$ CVaR, yielding Distributionally Robust Policy Optimization (DRPO). DRPO decouples what to learn (hard-filtered high-quality signals) from how to learn (adaptive trust-region optimization), and a variance-guided curriculum ensures stable, efficient learning and safe offline-to-online transitions. Empirical results on high-fidelity RecSim benchmarks show state-of-the-art performance, robustness to extreme noise, and strong generalization, suggesting a practical paradigm shift toward explicit data rejection in production RL systems.

Abstract

Policy-based Reinforcement Learning (RL) has established itself as the dominant paradigm in generative recommendation for optimizing sequential user interactions. However, when applied to offline historical logs, these methods suffer a critical failure: the dominance of low-quality data induces severe model collapse. We first establish the Divergence Theory of Repulsive Optimization, revealing that negative gradient updates inherently trigger exponential intensity explosion during off-policy training. This theory elucidates the inherent dilemma of existing methods, exposing their inability to reconcile variance reduction and noise imitation. To break this curse, we argue that the solution lies in rigorously identifying the latent high-quality distribution entangled within the noisy behavior policy. Accordingly, we reformulate the objective as an Optimistic Distributionally Robust Optimization (DRO) problem. Guided by this formulation, we propose Distributionally Robust Policy Optimization (DRPO). We prove that hard filtering is the exact solution to this DRO objective, enabling DRPO to optimally recover high-quality behaviors while strictly discarding divergence-inducing noise. Extensive experiments demonstrate that DRPO achieves state-of-the-art performance on mixed-quality recommendation benchmarks.

Figures (6)

• Figure 1: Collected from a large-scale industrial recommendation system, the eCPM statistics exhibit a strict Zipfian distribution. High-value signals are extremely sparse, while the vast majority of interactions constitute low-value noise.
• Figure 2: Verification of Divergence Theory. (a) Gradient Explosion: Negative gradients explode ($\approx 9\times$). (b) Structural Expansion: Both $\mu$ and $\sigma$ expand. Crucially, the repulsive imbalance on $\sigma$ (Ratio $\approx 58\times$) is far more severe than on $\mu$ (Ratio $\approx 2\times$).
• Figure 3: Robustness against Noise Injection. DRPO maintains performance as noise increases, whereas soft-weighting methods degrade significantly.
• Figure 4: Evolutionary Dynamics. DRPO seamlessly transitions from offline safeguarding to online exploration.
• Figure 5: Overview of RecSim. The system generates synthetic user-item interactions based on high-dimensional embedding clusters, featuring customizable noise injection and a mixed-strategy logging agent to replicate industrial data distributions.

Theorems & Definitions (10)

• Proposition 3.1: Geometry of the Score Function
• Theorem 3.2: Sign-Dependent Dynamics
• Corollary 3.3: Quadratic Gradient Explosion
• Theorem 3.4: Contraction-Induced Fragility
• Corollary 3.5: Asymptotic Signal-to-Noise Collapse
• Proposition 3.6: Translation Invariance of On-Policy Gradients
• Theorem 4.1: Optimality of Hard Filtering
• proof
• proof
• proof