Post-Training Denoising of User Profiles with LLMs in Collaborative Filtering Recommendation
Ervin Dervishaj, Maria Maistro, Tuukka Ruotsalo, Christina Lioma
TL;DR
The paper addresses the problem of noisy implicit feedback in collaborative filtering by proposing a post-training denoising method that leverages pretrained Large Language Models (LLMs) to refine user profiles without altering the CF model or training data. The method prompts the LLM with the user history $I_u$, a candidate item $j$, and its rank $r_j$ from the trained model $\mathcal{M}$, asking the LLM to remove an item $k$ so that $r_j' = \mathcal{M}(I_u\setminus k, j)$ improves, and accepts the denoised profile only if the rank improves; this can be done in zero-shot or few-shot modes with optional in-context examples. Experiments on MovieLens 1M, Yelp, and Amazon CDs & Vinyl using MultiVAE and four LLMs show consistent gains in ranking effectiveness, with up to $13\%$ improvements in $NDCG$ and similar gains in $HR$; several LLM configurations even surpass the upper-bound baselines without retraining the CF model. This work demonstrates that LLMs can be used as a cost-effective input-refinement step for CF, enabling practical denoising without data collection or model re-training, and they provide insights into how prompt design and profile characteristics shape effectiveness. Overall, the work paves a path for leveraging pretrained LLMs to systematically cleanse input signals in RSs, reducing reliance on in-training denoising and additional data.
Abstract
Implicit feedback -- the main data source for training Recommender Systems (RSs) -- is inherently noisy and has been shown to negatively affect recommendation effectiveness. Denoising has been proposed as a method for removing noisy implicit feedback and improving recommendations. Prior work has focused on in-training denoising, however this requires additional data, changes to the model architecture and training procedure or fine-tuning, all of which can be costly and data hungry. In this work, we focus on post-training denoising. Different from in-training denoising, post-training denoising does not involve changing the architecture of the model nor its training procedure, and does not require additional data. Specifically, we present a method for post-training denoising user profiles using Large Language Models (LLMs) for Collaborative Filtering (CF) recommendations. Our approach prompts LLMs with (i) a user profile (user interactions), (ii) a candidate item, and (iii) its rank as given by the CF recommender, and asks the LLM to remove items from the user profile to improve the rank of the candidate item. Experiments with a state-of-the-art CF recommender and 4 open and closed source LLMs in 3 datasets show that our denoising yields improvements up to 13% in effectiveness over the original user profiles. Our code is available at https://github.com/edervishaj/denoising-user-profiles-LLM.