RALLRec+: Retrieval Augmented Large Language Model Recommendation with Reasoning

TL;DR

RALLRec+ tackles two bottlenecks in retrieval-augmented LLM recommendations by creating a joint representation of textual item descriptions and collaborative signals, then refining retrieval with a time-aware reranker. In generation, it assesses reasoning LLMs for recommendation tasks and introduces knowledge-injected prompting plus consistency-based merging to fuse reasoning-enabled and general LLMs. The approach yields strong improvements on three real-world CTR datasets, showing that retrieval augmentation boosts reasoning, and that consistent, domain-aware prompting enhances final predictions. The work highlights the potential of combining representation learning with reasoning modules to improve interpretability and performance in LLM-powered recommender systems.

Abstract

Large Language Models (LLMs) have been integrated into recommender systems to enhance user behavior comprehension. The Retrieval Augmented Generation (RAG) technique is further incorporated into these systems to retrieve more relevant items and improve system performance. However, existing RAG methods have two shortcomings. \textit{(i)} In the \textit{retrieval} stage, they rely primarily on textual semantics and often fail to incorporate the most relevant items, thus constraining system effectiveness. \textit{(ii)} In the \textit{generation} stage, they lack explicit chain-of-thought reasoning, further limiting their potential. In this paper, we propose Representation learning and \textbf{R}easoning empowered retrieval-\textbf{A}ugmented \textbf{L}arge \textbf{L}anguage model \textbf{Rec}ommendation (RALLRec+). Specifically, for the retrieval stage, we prompt LLMs to generate detailed item descriptions and perform joint representation learning, combining textual and collaborative signals extracted from the LLM and recommendation models, respectively. To account for the time-varying nature of user interests, we propose a simple yet effective reranking method to capture preference dynamics. For the generation phase, we first evaluate reasoning LLMs on recommendation tasks, uncovering valuable insights. Then we introduce knowledge-injected prompting and consistency-based merging approach to integrate reasoning LLMs with general-purpose LLMs, enhancing overall performance. Extensive experiments on three real world datasets validate our method's effectiveness.

Figures (9)

• Figure 1: Example of the response generated by general LLM and reasoning LLM.
• Figure 2: Comparison of response length w.r.t. accuracy in reasoning LLMs.
• Figure 3: Comparison of response consistency w.r.t. accuracy in reasoning LLMs.
• Figure 4: RALLRec+ with representation learning enhanced retrieval and reasoning enhanced generation.
• Figure 5: Comparison of textual descriptions with fixed template (upper) and automatic generation (blow).