Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization
Yu Hou, Hua Li, Ha Young Kim, Won-Yong Shin
TL;DR
This work addresses the high computational cost and data sensitivity of fine-tuning diffusion-based recommender systems by introducing ReFiT, an RL-aided fine-tuning framework that operates on pre-trained diffusion models without architectural changes. By formulating the denoising process as an MDP and employing a collaborative-signal-aware reward (RACS), ReFiT guides policy optimization via REINFORCE to directly maximize the log-likelihood of observed interactions, achieving substantial gains (e.g., up to 36.3% in NDCG@20) with linear complexity in the number of users and items. Theoretical analysis links ReFiT to exact log-likelihood optimization and demonstrates scalability, while empirical results across standard CF, sequential, social, and POI diffusion-based tasks confirm improvements over strong baselines and notable efficiency gains. The approach offers practical impact for scalable, personalized diffusion-based recommendations and opens avenues for richer reward designs, including future integration with language-model guidance.
Abstract
Diffusion models recently emerged as a powerful paradigm for recommender systems, offering state-of-the-art performance by modeling the generative process of user-item interactions. However, training such models from scratch is both computationally expensive and yields diminishing returns once convergence is reached. To remedy these challenges, we propose ReFiT, a new framework that integrates Reinforcement learning (RL)-based Fine-Tuning into diffusion-based recommender systems. In contrast to prior RL approaches for diffusion models depending on external reward models, ReFiT adopts a task-aligned design: it formulates the denoising trajectory as a Markov decision process (MDP) and incorporates a collaborative signal-aware reward function that directly reflects recommendation quality. By tightly coupling the MDP structure with this reward signal, ReFiT empowers the RL agent to exploit high-order connectivity for fine-grained optimization, while avoiding the noisy or uninformative feedback common in naive reward designs. Leveraging policy gradient optimization, ReFiT maximizes exact log-likelihood of observed interactions, thereby enabling effective post hoc fine-tuning of diffusion recommenders. Comprehensive experiments on wide-ranging real-world datasets demonstrate that the proposed ReFiT framework (a) exhibits substantial performance gains over strong competitors (up to 36.3% on sequential recommendation), (b) demonstrates strong efficiency with linear complexity in the number of users or items, and (c) generalizes well across multiple diffusion-based recommendation scenarios. The source code and datasets are publicly available at https://anonymous.4open.science/r/ReFiT-4C60.