LLM as Explainable Re-Ranker for Recommendation System
Yaqi Wang, Haojia Sun, Shuting Zhang
TL;DR
The paper tackles the explainability and bias issues in traditional recommender systems by introducing an LLM-based re-ranker that operates in tandem with conventional models. It proposes a two-stage training pipeline (SFT followed by DPO with RPO) alongside bootstrapping and self-consistency to mitigate position and popularity biases while generating user-friendly explanations. A carefully constructed dataset and extensive experiments demonstrate improved NDCG and explainability, particularly for weaker base models, with statistical validation. The work highlights the potential of integrating LLMs into ranking to produce transparent, fair, and effective recommendations, while noting limitations in leveraging graph information and dataset quality for further improvements.
Abstract
The application of large language models (LLMs) in recommendation systems has recently gained traction. Traditional recommendation systems often lack explainability and suffer from issues such as popularity bias. Previous research has also indicated that LLMs, when used as standalone predictors, fail to achieve accuracy comparable to traditional models. To address these challenges, we propose to use LLM as an explainable re-ranker, a hybrid approach that combines traditional recommendation models with LLMs to enhance both accuracy and interpretability. We constructed a dataset to train the re-ranker LLM and evaluated the alignment between the generated dataset and human expectations. Leveraging a two-stage training process, our model significantly improved NDCG, a key ranking metric. Moreover, the re-ranker outperformed a zero-shot baseline in ranking accuracy and interpretability. These results highlight the potential of integrating traditional recommendation models with LLMs to address limitations in existing systems and pave the way for more explainable and fair recommendation frameworks.