Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHF

Keertana Chidambaram, Sanath Kumar Krishnamurthy, Qiuling Xu, Ko-Jen Hsiao, Moumita Bhattacharya

TL;DR

It is shown that temperature $\lambda$ explicitly and quantifiably controls the robustness-improvement tradeoff, providing practitioners with a single interpretable regularization hyperparameter with theoretical grounding.

Abstract

Aligning generative recommender systems to user preferences via post-training is critical for closing the gap between next-item prediction and actual recommendation quality. Existing post-training methods are ill-suited for production-scale systems: RLHF methods reward hack due to noisy user feedback and unreliable reward models, offline RL alternatives require propensity scores that are unavailable, and online interaction is infeasible. We identify exponential reward-weighted SFT with weights $w = \exp(r/λ)$ as uniquely suited to this setting, and provide the theoretical and empirical foundations that explain why. By optimizing directly on observed rewards without querying a learned reward model, the method is immune to reward hacking, requires no propensity scores, and is fully offline. We prove the first policy improvement guarantees for this setting under noisy rewards, showing that the gap scales only logarithmically with catalog size and remains informative even for large item catalogs. Crucially, we show that temperature $λ$ explicitly and quantifiably controls the robustness-improvement tradeoff, providing practitioners with a single interpretable regularization hyperparameter with theoretical grounding. Experiments on three open-source and one proprietary dataset against four baselines confirm that exponential reward weighting is simple, scalable, and consistently outperforms RLHF-based alternatives.

Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHF

TL;DR

It is shown that temperature explicitly and quantifiably controls the robustness-improvement tradeoff, providing practitioners with a single interpretable regularization hyperparameter with theoretical grounding.

Abstract

Aligning generative recommender systems to user preferences via post-training is critical for closing the gap between next-item prediction and actual recommendation quality. Existing post-training methods are ill-suited for production-scale systems: RLHF methods reward hack due to noisy user feedback and unreliable reward models, offline RL alternatives require propensity scores that are unavailable, and online interaction is infeasible. We identify exponential reward-weighted SFT with weights as uniquely suited to this setting, and provide the theoretical and empirical foundations that explain why. By optimizing directly on observed rewards without querying a learned reward model, the method is immune to reward hacking, requires no propensity scores, and is fully offline. We prove the first policy improvement guarantees for this setting under noisy rewards, showing that the gap scales only logarithmically with catalog size and remains informative even for large item catalogs. Crucially, we show that temperature explicitly and quantifiably controls the robustness-improvement tradeoff, providing practitioners with a single interpretable regularization hyperparameter with theoretical grounding. Experiments on three open-source and one proprietary dataset against four baselines confirm that exponential reward weighting is simple, scalable, and consistently outperforms RLHF-based alternatives.
Paper Structure (15 sections, 3 theorems, 43 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 15 sections, 3 theorems, 43 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Background
  4. Problem Setting
  5. Exponential Reward-Weighted Policy Optimization
  6. Theoretical Analysis
  7. Extension to the MDP Setting
  8. Experiments
  9. Limitations
  10. Proofs
  11. Proposition \ref{['prop:monotonic']} Proof
  12. Theorem \ref{['thm:noisy_improvement']} Proof
  13. Theorem \ref{['thm:temp_bound']} Proof
  14. Experiments
  15. Data Distribution

Key Result

Proposition 4.1

Let $\pi^*_{\lambda}(a|s) \propto \pi_\beta(a|s) \exp\left(\frac{r^*(s,a)}{\lambda}\right)$. Then in the contextual bandit setting:

Figures (4)

  • Figure 1: Comparison of policy learning methods along two axes: best achievable policy performance and dependency on a good reward model.
  • Figure 2: Figure showing the dramatic collapse of PPO and DPO due to reward-hacking.
  • Figure 3: NDCG@10 for different values of $\lambda$ for all three datasets.
  • Figure 4: Distribution of ratings in all three test datasets.

Theorems & Definitions (6)

  • Proposition 4.1: Monotonic Policy Improvement
  • Theorem 4.3: Policy Improvement Under Noisy Rewards
  • Theorem 4.4: Temperature-Dependent Bound
  • proof
  • proof
  • proof