Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPRec: Self-Play to Debias LLM-based Recommendation

Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, Xiangnan He

TL;DR

SPRec introduces a self-play training framework for LLM-based recommender systems to counteract bias amplification inherent in Direct Preference Optimization (DPO). By alternating Supervised Fine-Tuning on positive offline data with a DPO step that uses the model’s own previous outputs as negative samples, SPRec adaptively suppresses biased items while preserving alignment with user preferences. Empirical results across four real-world datasets show improvements in accuracy, diversity, and fairness over SFT- and DPO-based baselines, with ablations confirming the value of self-play negative sampling. The work demonstrates a practical path toward self-correcting, fairer LRSs and suggests future directions for long-horizon alignment and bias mitigation during SFT as well as policy-level optimization.

Abstract

Large language models (LLMs) have attracted significant attention in recommendation systems. Current work primarily applies supervised fine-tuning (SFT) to adapt the model for recommendation tasks. However, SFT on positive examples only limits the model's ability to align with user preference. To address this, researchers recently introduced Direct Preference Optimization (DPO), which explicitly aligns LLMs with user preferences using offline preference ranking data. However, we found that DPO inherently biases the model towards a few items, exacerbating the filter bubble issue and ultimately degrading user experience. In this paper, we propose SPRec, a novel self-play framework designed to mitigate over-recommendation and improve fairness without requiring additional data or manual intervention. In each self-play iteration, the model undergoes an SFT step followed by a DPO step, treating offline interaction data as positive samples and the predicted outputs from the previous iteration as negative samples. This effectively re-weights the DPO loss function using the model's logits, adaptively suppressing biased items. Extensive experiments on multiple real-world datasets demonstrate SPRec's effectiveness in enhancing recommendation accuracy and fairness. The implementation is available via https://github.com/RegionCh/SPRec

SPRec: Self-Play to Debias LLM-based Recommendation

TL;DR

SPRec introduces a self-play training framework for LLM-based recommender systems to counteract bias amplification inherent in Direct Preference Optimization (DPO). By alternating Supervised Fine-Tuning on positive offline data with a DPO step that uses the model’s own previous outputs as negative samples, SPRec adaptively suppresses biased items while preserving alignment with user preferences. Empirical results across four real-world datasets show improvements in accuracy, diversity, and fairness over SFT- and DPO-based baselines, with ablations confirming the value of self-play negative sampling. The work demonstrates a practical path toward self-correcting, fairer LRSs and suggests future directions for long-horizon alignment and bias mitigation during SFT as well as policy-level optimization.

Abstract

Large language models (LLMs) have attracted significant attention in recommendation systems. Current work primarily applies supervised fine-tuning (SFT) to adapt the model for recommendation tasks. However, SFT on positive examples only limits the model's ability to align with user preference. To address this, researchers recently introduced Direct Preference Optimization (DPO), which explicitly aligns LLMs with user preferences using offline preference ranking data. However, we found that DPO inherently biases the model towards a few items, exacerbating the filter bubble issue and ultimately degrading user experience. In this paper, we propose SPRec, a novel self-play framework designed to mitigate over-recommendation and improve fairness without requiring additional data or manual intervention. In each self-play iteration, the model undergoes an SFT step followed by a DPO step, treating offline interaction data as positive samples and the predicted outputs from the previous iteration as negative samples. This effectively re-weights the DPO loss function using the model's logits, adaptively suppressing biased items. Extensive experiments on multiple real-world datasets demonstrate SPRec's effectiveness in enhancing recommendation accuracy and fairness. The implementation is available via https://github.com/RegionCh/SPRec

Paper Structure

This paper contains 36 sections, 1 theorem, 7 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. LLMs for Recommendation
  4. Biases in Recommender Systems
  5. Self-Play Mechanism
  6. Preliminary
  7. Supervised Fine-tuning (SFT)
  8. Direct Preference Optimization (DPO)
  9. Evaluating Bias via Distribution Alignment
  10. Problem of DPO: Amplify Popularity Bias
  11. Empirical Analysis
  12. Theoretical Analysis
  13. Method
  14. Solution: Suppress Biases through Self-Play
  15. Architecture of SPRec
  16. ...and 21 more sections

Key Result

Theorem 1

The optimal policy $\pi_\theta^*(\cdot|x)$ for the DPO loss defined in Eq. (eq:dpo_new) is given by:

Figures (7)

  • Figure 1: Homogeneity issues in LLM-based recommendation results caused by token-level and item-level biases.
  • Figure 2: Distribution of cold-start recommendation results. Group 0: least popular, group 4: most popular.
  • Figure 3: Illustration of SFT, DPO, and SPRec in LLM-based recommendations. (a) SFT generates mass-covering results but retains inherent biases. (b) DPO amplifies these biases by over-representing certain items. (c) SPRec mitigates bias through self-play, leveraging model outputs as negative samples to achieve balanced recommendations.
  • Figure 4: Comparison of models across genres on Group Unfairness (GU) in top-$1$ recommendation.
  • Figure 5: Performance on the MovieLens dataset across different ablation experiments.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Theorem 1