LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations

TL;DR

This work presents LLM4Rerank, a novel framework that uses a fully connected graph of specialized nodes to integrate accuracy, diversity, and fairness in the LLM-driven reranking stage of recommender systems. It introduces a generic node structure, a historical reranking pool, and a Goal-driven automatic transition mechanism to enable personalized, multi-aspect reranking at scale. Empirical results on three public datasets show LLM4Rerank achieving superior balance across criteria compared to strong baselines, with ablations highlighting the importance of the pool and adaptive routing. The approach offers a flexible, extensible path toward practical, aspect-aware reranking, with potential impact on real-world recommender systems by enhancing personalization and fairness while maintaining performance.

Abstract

Reranking is a critical component in recommender systems, playing an essential role in refining the output of recommendation algorithms. Traditional reranking models have focused predominantly on accuracy, but modern applications demand consideration of additional criteria such as diversity and fairness. Existing reranking approaches often fail to harmonize these diverse criteria effectively at the model level. Moreover, these models frequently encounter challenges with scalability and personalization due to their complexity and the varying significance of different reranking criteria in diverse scenarios. In response, we introduce a comprehensive reranking framework enhanced by LLM, designed to seamlessly integrate various reranking criteria while maintaining scalability and facilitating personalized recommendations. This framework employs a fully connected graph structure, allowing the LLM to simultaneously consider multiple aspects such as accuracy, diversity, and fairness through a coherent Chain-of-Thought (CoT) process. A customizable input mechanism is also integrated, enabling the tuning of the language model's focus to meet specific reranking needs. We validate our approach using three popular public datasets, where our framework demonstrates superior performance over existing state-of-the-art reranking models in balancing multiple criteria.

Figures (5)

• Figure 1: Ranking and reranking process in recommendations.
• Figure 2: Overall structure of LLM4Rerank. Inputs are first directed to the "Accuracy" node, which initiates an automatic reranking process in (a). Nodes (b) with varying colors represent distinct aspects or functional steps, guiding the LLM in its deliberations with historical information in (c). A complete reranking process is considered finished once LLM reaches the "Stop" node. For simplicity, items in this figure are represented by their IDs, and the detailed descriptions are hidden.
• Figure 7: Case study of LLM4Rerank on ML-1M dataset. The figure shows the most common paths for LLM4Rerank under the two "Goals". The evaluation is based on the average result on the specific path.
• Figure 8: Hyper-parameter analysis of the "candidate item number" with LLM4Rerank-ADF on ML-1M dataset.
• Figure 9: Example prompt template of the novelty node.