CoRA: Collaborative Information Perception by Large Language Model's Weights for Recommendation

TL;DR

A new paradigm, Collaborative LoRA (CoRA), is proposed with a collaborative filtering model to extract user and item embeddings and inject them into a set number of learnable queries, enabling LLM to perceive the collaborative signals and generate personalized recommendations without fine-tuning or extra collaborative tokens in prompts.

Abstract

Involving collaborative information in Large Language Models (LLMs) is a promising technique for adapting LLMs for recommendation. Existing methods achieve this by concatenating collaborative features with text tokens into a unified sequence input and then fine-tuning to align these features with LLM's input space. Although effective, in this work, we identify two limitations when adapting LLMs to recommendation tasks, which hinder the integration of general knowledge and collaborative information, resulting in sub-optimal recommendation performance. (1) Fine-tuning LLM with recommendation data can undermine its inherent world knowledge and fundamental competencies, which are crucial for interpreting and inferring recommendation text. (2) Incorporating collaborative features into textual prompts disrupts the semantics of the original prompts, preventing LLM from generating appropriate outputs. In this paper, we propose a new paradigm, \textbf{Co}llaborative \textbf{Lo}RA (CoRA), with a collaborative query generator. Rather than input space alignment, this method aligns collaborative information with LLM's parameter space, representing them as incremental weights to update LLM's output. This way, LLM perceives collaborative information without altering its general knowledge and text inference capabilities. Specifically, we employ a collaborative filtering model to extract user and item embeddings and inject them into a set number of learnable queries. We then convert collaborative queries into collaborative weights with low-rank properties and merge the collaborative weights into LLM's weights, enabling LLM to perceive the collaborative signals and generate personalized recommendations without fine-tuning or extra collaborative tokens in prompts. Extensive experiments confirm that CoRA effectively integrates collaborative information into LLM, enhancing recommendation performance.

Figures (6)

• Figure 1: The performance of Vicuna-7B before and after fine-tuning on Amazon-Book using the prompt in TALLRec. The EM/ROUGE-L scores of generated answers on datasets represent various general and recommendation abilities.
• Figure 2: Collaborative features interfering with LLM's understanding of textual prompts. We use pre-trained Vicuna-7B as the ground truth. Our method avoids this interference.
• Figure 3: (a) Architecture of the LLM's Decoder Block. (b) Details of the multi-head self-attention module.
• Figure 4: Model architecture overview of our CoRA. The left path extracts user and item embeddings using a CF model and generates collaborative queries. The right path fills the text fields in the prompt template, introducing textual descriptions for inference. Finally, the collaborative queries are projected into the LLM's parameter space and merged into the LLM's weights, enabling the LLM to perceive collaborative information without any fine-tuning or extra tokens in textual prompts.
• Figure 5: Performance comparison in warm and cold scenarios on Amazon-Book and ML-1M. The left and right y-axis are AUC and UAUC, respectively.