Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMInit: A Free Lunch from Large Language Models for Selective Initialization of Recommendation

Weizhi Zhang, Liangwei Yang, Wooseong Yang, Henry Peng Zou, Yuqing Liu, Ke Xu, Sourav Medya, Philip S. Yu

TL;DR

This work tackles cold-start and data-sparsity challenges in collaborative filtering by leveraging pretrained LLM embeddings through selective initialization. It introduces LLMInit, which uses uniform, random, or variance-based sampling to distill high-dimensional LLM representations into compact item embeddings while aggregating item signals to form robust user embeddings. Across four real-world Amazon datasets and multiple CF backbones, LLMInit consistently improves ranking metrics, with variance-based selection delivering the strongest gains and maintaining a lightweight footprint (~2M parameters). The approach offers a scalable, industry-friendly alternative to full LLM-based recommendations and is released as open-source for adoption and further research.

Abstract

Collaborative filtering (CF) is widely adopted in industrial recommender systems (RecSys) for modeling user-item interactions across numerous applications, but often struggles with cold-start and data-sparse scenarios. Recent advancements in pre-trained large language models (LLMs) with rich semantic knowledge, offer promising solutions to these challenges. However, deploying LLMs at scale is hindered by their significant computational demands and latency. In this paper, we propose a novel and scalable LLM-RecSys framework, LLMInit, designed to integrate pretrained LLM embeddings into CF models through selective initialization strategies. Specifically, we identify the embedding collapse issue observed when CF models scale and match the large embedding sizes in LLMs and avoid the problem by introducing efficient sampling methods, including, random, uniform, and variance-based selections. Comprehensive experiments conducted on multiple real-world datasets demonstrate that LLMInit significantly improves recommendation performance while maintaining low computational costs, offering a practical and scalable solution for industrial applications. To facilitate industry adoption and promote future research, we provide open-source access to our implementation at https://github.com/DavidZWZ/LLMInit.

LLMInit: A Free Lunch from Large Language Models for Selective Initialization of Recommendation

TL;DR

This work tackles cold-start and data-sparsity challenges in collaborative filtering by leveraging pretrained LLM embeddings through selective initialization. It introduces LLMInit, which uses uniform, random, or variance-based sampling to distill high-dimensional LLM representations into compact item embeddings while aggregating item signals to form robust user embeddings. Across four real-world Amazon datasets and multiple CF backbones, LLMInit consistently improves ranking metrics, with variance-based selection delivering the strongest gains and maintaining a lightweight footprint (~2M parameters). The approach offers a scalable, industry-friendly alternative to full LLM-based recommendations and is released as open-source for adoption and further research.

Abstract

Collaborative filtering (CF) is widely adopted in industrial recommender systems (RecSys) for modeling user-item interactions across numerous applications, but often struggles with cold-start and data-sparse scenarios. Recent advancements in pre-trained large language models (LLMs) with rich semantic knowledge, offer promising solutions to these challenges. However, deploying LLMs at scale is hindered by their significant computational demands and latency. In this paper, we propose a novel and scalable LLM-RecSys framework, LLMInit, designed to integrate pretrained LLM embeddings into CF models through selective initialization strategies. Specifically, we identify the embedding collapse issue observed when CF models scale and match the large embedding sizes in LLMs and avoid the problem by introducing efficient sampling methods, including, random, uniform, and variance-based selections. Comprehensive experiments conducted on multiple real-world datasets demonstrate that LLMInit significantly improves recommendation performance while maintaining low computational costs, offering a practical and scalable solution for industrial applications. To facilitate industry adoption and promote future research, we provide open-source access to our implementation at https://github.com/DavidZWZ/LLMInit.

Paper Structure

This paper contains 19 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Definition
  4. Related Work
  5. Method
  6. Selective Item Embedding Initialization
  7. Uniform Selection (LLMInit-Uni)
  8. Random Selection (LLMInit-Rand)
  9. Variance Selection (LLMInit-Var)
  10. Aggregated User Embedding Initialization
  11. Experiments
  12. Experimental Set Up
  13. Performance Comparison
  14. Efficiency and Perforamance Comparisons of LLM-based RecSys
  15. Cold-Start Recommendation
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An investigation of the embedding collapse issue of CF model LightGCN he2020lightgcn in two Amazon-review datasets. Increasing embedding size with exponentially growing parameters finally leads to performance degradation.
  • Figure 2: An Illustration of LLMInit framework including contextual LLM input, three types of selective item embedding initialization strategies, and the user embedding aggregation operation.
  • Figure 3: Performance improvement comparison in non-strict cold-start scenarios on Amazon-Beauty, where half of the observed interactions for each user are removed.