LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application

TL;DR

This work designs a twin-tower structure supervised by the recommendation task and tailored for practical industrial application that synergizes open-world knowledge with collaborative knowledge and addresses computational complexity concerns by utilizing pretrained LLMs as item encoders and freezing LLM parameters to avoid catastrophic forgetting and preserve open-world knowledge.

Abstract

Contemporary recommendation systems predominantly rely on ID embedding to capture latent associations among users and items. However, this approach overlooks the wealth of semantic information embedded within textual descriptions of items, leading to suboptimal performance and poor generalizations. Leveraging the capability of large language models to comprehend and reason about textual content presents a promising avenue for advancing recommendation systems. To achieve this, we propose an Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework that synergizes open-world knowledge with collaborative knowledge. We address computational complexity concerns by utilizing pretrained LLMs as item encoders and freezing LLM parameters to avoid catastrophic forgetting and preserve open-world knowledge. To bridge the gap between the open-world and collaborative domains, we design a twin-tower structure supervised by the recommendation task and tailored for practical industrial application. Through experiments on the real large-scale industrial dataset and online A/B tests, we demonstrate the efficacy of our approach in industry application. We also achieve state-of-the-art performance on six Amazon Review datasets to verify the superiority of our method.

Figures (6)

• Figure 1: Illustration of "Rec-to-LLM" and "LLM-to-Rec" approaches. Previous methods textualize recommendation data to natural language conversations, which are then fed into LLM to obtain text predictions. In contrast, our method, LEARN, transforms item text information into LLM embedding, which is projected into the collaborative domain to achieve alignment with industry recommendation tasks.
• Figure 2: Illustration of our Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework. The LEARN framework employs a twin-tower architecture comprising a user tower and an item tower. The user tower processes history interactions to generate user embeddings $E^{user}$, while the item tower handles target interactions to produce item embeddings $E^{item}$. User Tower and Item Tower (a) leverage the causal attention mechanism. Item Tower (b) adopts a self-attention mechanism. Without the preference alignment module, Item Tower (c) directly utilizes the content embedding as the item embedding.
• Figure 3: Illustration of the Content Extraction (CEX ) module and Preference ALignment (PAL) module. The CEX module utilizes a pretrained LLM to generate content embeddings from item text descriptions. The PAL module takes these content embeddings and projects them from the open-world domain into the collaborative domain embeddings used in the online recommender.
• Figure 4: Performance comparison with text-only SOTA methods under zero-shot setting.
• Figure 5: Illustration of online ranking model structure.