Safe, Efficient, and Robust Reinforcement Learning for Ranking and Diffusion Models
Shashank Gupta
TL;DR
This work investigates how to make reinforcement learning for ranking and generative modeling safe, sample-efficient, and robust using a unified contextual-bandit perspective. It introduces exposure-based generalization bounds and a safe counterfactual learning-to-rank framework to guarantee that new policies do not underperform the production system, even with sparse feedback. It then unifies off-policy estimators through an optimal baseline correction, deriving closed-form solutions that minimize both gradient and value variance, and demonstrates substantial variance reductions in off-policy learning and evaluation. Finally, it studies the efficiency–effectiveness trade-off in diffusion-model fine-tuning, proposing LOOP, which blends REINFORCE baseline variance reduction with PPO clipping to achieve PPO-like performance with reduced computational costs. Collectively, the thesis provides practical safety guarantees, data-efficient learning from logs, and scalable RL techniques for both ranking systems and large diffusion models, with open-source implementations to support reproducibility.
Abstract
This dissertation investigates how reinforcement learning (RL) methods can be designed to be safe, sample-efficient, and robust. Framed through the unifying perspective of contextual-bandit RL, the work addresses two major application domains - ranking and recommendation, and text-to-image diffusion models. The first part of the thesis develops theory and algorithms for safe deployment in ranking systems. An exposure-based generalisation bound is derived, leading to a counterfactual risk-minimisation objective whose solution is guaranteed not to underperform the logging policy, even with sparse feedback. This guarantee is extended to doubly robust estimators, enabling safety even under adversarial or misspecified user models and offering practitioners explicit control over permissible utility loss. The second part turns to single-action bandits, where various off-policy estimators are unified within a baseline-correction framework. A closed-form optimal baseline is proposed and shown to minimise both evaluation and policy-gradient variance, thereby improving off-policy learning reliability. The final part examines the trade-offs between efficiency and effectiveness in generative RL. A systematic study of PPO and REINFORCE motivates the Leave-One-Out PPO (LOOP) algorithm, which combines multiple diffusion trajectories with a REINFORCE-style baseline inside PPO's clipped objective. LOOP achieves PPO-level sample efficiency while producing generations that align more faithfully with textual attributes.