GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networks

TL;DR

This work tackles exposure bias in generative recommendations by framing fine-tuning as multi-step generation and applying Generative Flow Networks (GFlowNets). It introduces GFlowGR, which combines a four-strategy trajectory sampler, a multi-signal reward model, and DB/TB flow-matching objectives to jointly train LLMs with augmented trajectories and standard next-token supervision. Empirical results across three real datasets and two backbones show consistent improvements over SFT and RL-based baselines, with additional offline and production deployments in Taobao demonstrating practical gains (e.g., increased revenue). The approach advances GR by enabling diverse, high-reward trajectory generation and leveraging collaborative knowledge during training, offering a robust solution to exposure bias in large-scale, real-world recommender systems.

Abstract

Generative recommendations (GR), which usually include item tokenizers and generative Large Language Models (LLMs), have demonstrated remarkable success across a wide range of scenarios. The majority of existing research efforts primarily concentrate on developing powerful item tokenizers or advancing LLM decoding strategies to attain superior performance. However, the critical fine-tuning step in GR frameworks, which is essential for adapting LLMs to recommendation data, remains largely unexplored. Current approaches predominantly rely on either the next-token prediction loss of supervised fine-tuning (SFT) or recommendationspecific direct preference optimization (DPO) strategies. Both methods ignore the exploration of possible positive unobserved samples, which is commonly referred to as the exposure bias problem. To mitigate this problem, this paper treats the GR as a multi-step generation task and constructs a GFlowNets-based fine-tuning framework (GFlowGR). The proposed framework integrates collaborative knowledge from traditional recommender systems to create an adaptive trajectory sampler and a comprehensive reward model. Leveraging the diverse generation property of GFlowNets, along with sampling and heuristic weighting techniques, GFlowGR emerges as a promising approach to mitigate the exposure bias problem. Extensive empirical results on two real-world datasets and with two different GR backbones highlight the effectiveness and robustness of GFlowGR.

Figures (4)

• Figure 1: Overview of GR and GFlowNets, with the length of item identifiers $L=3$. (a) A process for constucting the GR framwork. (b) Generative flow structure is exemplified with a set of user interaction, which typically includes positive feedback ("Clicked"), impression ("Presented"), and unexposed items ("unpresented").
• Figure 2: Framework of GFlowGR. The visualized example shows a the learning on a $N$-item set illustrated with two items among the set, including a positive sample (blue polo shirt) and an unpresented sample (yellow undershirt). Forward probabilities ($\mathbb{P}_F$) for both trajectories are displayed (e.g., $0.9, 0.8, 0.83$ for $\tau_1$ ), while estimated flow values ($\mathcal{F}$) are omitted for brevity. The bluesky shaded area presents a example of incorporating collaborative models (CM).
• Figure 3: Parameter study on $N$ and $\lambda$ with TIGER on Beauty. The red dashed line denotes the performance of SFT, which falls outside the plotted range for the figures corresponding to $N$.
• Figure 4: Deployed business scenarios in Taobao, illustrated with the query "Dress" in the top left red box. Advertisements are indicated by blue boxes on the bottom right of each product. Due to privacy policy, sensitive information has been obfuscated.