Enhance Large Language Models as Recommendation Systems with Collaborative Filtering

TL;DR

This paper tackles the limitation of non-tuning LLM-based recommenders by introducing Critic-LLM-RS, which couples a frozen LLM with a separate Recommendation Critic that embodies collaborative filtering. The Critic provides targeted critiques to refine the LLM's initial recommendations, enabling CF-driven improvements without full model fine-tuning. Empirical evaluation on Movies and Books demonstrates superior performance over baselines and even some fine-tuning approaches, with multiple feedback loops offering diminishing returns. The approach leverages LLM knowledge while grounding recommendations in user-item interactions, yielding efficient, plug-in enhancements for large-scale recommender systems.

Abstract

As powerful tools in Natural Language Processing (NLP), Large Language Models (LLMs) have been leveraged for crafting recommendations to achieve precise alignment with user preferences and elevate the quality of the recommendations. The existing approaches implement both non-tuning and tuning strategies. Compared to following the tuning strategy, the approaches following the non-tuning strategy avoid the relatively costly, time-consuming, and expertise-requiring process of further training pre-trained LLMs on task-specific datasets, but they suffer the issue of not having the task-specific business or local enterprise knowledge. To the best of our knowledge, none of the existing approaches following the non-tuning strategy explicitly integrates collaborative filtering, one of the most successful recommendation techniques. This study aims to fill the gap by proposing critique-based LLMs as recommendation systems (Critic-LLM-RS). For our purpose, we train a separate machine-learning model called Critic that implements collaborative filtering for recommendations by learning from the interactions between many users and items. The Critic provides critiques to LLMs to significantly refine the recommendations. Extensive experiments have verified the effectiveness of Critic-LLM-RS on real datasets.

Figures (10)

• Figure 1: Critic-LLM-RS Architecture
• Figure 2: Evaluation on Books with Oracle.
• Figure 3: Evaluation on Movies with Oracle.
• Figure 4: Evaluation on Books with a candidate set.
• Figure 5: Evaluation on Movies with a candidate set.