Collaborative Retrieval for Large Language Model-based Conversational Recommender Systems

TL;DR

CRAG tackles the challenge of leveraging collaborative filtering within black-box LLM-based conversational recommender systems by introducing context-aware collaborative retrieval and a two-step reflection pipeline. It links dialogue to items via LLM-based entity extraction and bi-level matching, augments CF signals through context-aware retrieval using an adapted EASE objective, and mitigates LLM bias with reflect-and-rerank to produce quality top-N recommendations. Across Reddit-v2 and Redial, CRAG consistently outperforms zero-shot LLMs and Naive-RAG baselines, with the most pronounced gains for recently released items, and ablations demonstrate the necessity of both reflection stages. The work also provides a refined Reddit-v2 dataset and releases code/data, offering a practical benchmark and pathway for advancing LLM+CF CRS research.

Abstract

Conversational recommender systems (CRS) aim to provide personalized recommendations via interactive dialogues with users. While large language models (LLMs) enhance CRS with their superior understanding of context-aware user preferences, they typically struggle to leverage behavioral data, which have proven to be important for classical collaborative filtering (CF)-based approaches. For this reason, we propose CRAG, Collaborative Retrieval Augmented Generation for LLM-based CRS. To the best of our knowledge, CRAG is the first approach that combines state-of-the-art LLMs with CF for conversational recommendations. Our experiments on two publicly available movie conversational recommendation datasets, i.e., a refined Reddit dataset (which we name Reddit-v2) as well as the Redial dataset, demonstrate the superior item coverage and recommendation performance of CRAG, compared to several CRS baselines. Moreover, we observe that the improvements are mainly due to better recommendation accuracy on recently released movies. The code and data are available at https://github.com/yaochenzhu/CRAG.

Figures (14)

• Figure 1: An example for conversational recommendations, with items and relevant context highlighted in user query.
• Figure 2: Overview of CRAG for CRS and its three components: (i) LLM-based entity link; (ii) context-aware collaborative retrieval, and (iii) recommendation with reflect and rerank. The reflection steps are emphasized in green arrows. The sub- and super-script for different task-specific prompt $T$, format instruction $F$, and item list $\mathcal{I}$ are omitted for simplicity.
• Figure 3: Comparison of zero-shot LLM on the Reddit-v2 dataset and the one with randomly replaced items.
• Figure 4: The influence of the number of items in the raw collaborative retrieval $K$ (depicted in different color) on the recommendation performance of CRAG-nR12, CRAG-nR2, and CRAG. X-axis denotes the recall evaluated at top-$M$ positions.
• Figure 5: Comparison across different CRAG variants.