Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendation

Jiachen Zhu, Jianghao Lin, Xinyi Dai, Bo Chen, Rong Shan, Jieming Zhu, Ruiming Tang, Yong Yu, Weinan Zhang

TL;DR

RecLoRA tackles lifelong personalized LLM-based recommendation by introducing a per-user, parameter-efficient adaptation mechanism (Personalized LoRA) coupled with a soft-routing gate, a Long-Short Modality Retriever to balance history length across modalities, and a Few2Many magnifier that leverages a full-space conventional recommender to guide few-shot tuning of the LLM. The approach enables per-user LoRA customization, broader exposure to the training space without expanding fine-tuning cost, and efficient lifelong user modeling. Empirical results on three public datasets show RecLoRA outperforms strong baselines and demonstrates notable gains in few-shot settings, with ablations confirming the importance of all components. The work advances practical deployment of LLM-based recommender systems by addressing personalization, sequence length, and scalability together, with potential implications for real-world large-scale recommendation tasks.

Abstract

We primarily focus on the field of large language models (LLMs) for recommendation, which has been actively explored recently and poses a significant challenge in effectively enhancing recommender systems with logical reasoning abilities and open-world knowledge. Current mainstream efforts mainly center around injecting personalized information from recommendation models into LLMs by customizing input templates or aligning representations between semantic and recommendation spaces at the prediction layer. However, they face three significant limitations: (1) LoRA is mostly used as a core component in existing works, but personalization is not well established in LoRA parameters as the LoRA matrix shared by every user may not cater to different users' characteristics, leading to suboptimal performance. (2) Although lifelong personalized behavior sequences are ideal for personalization, their use raises effectiveness and efficiency issues since LLMs require escalating training and inference time to extend text lengths. (3) Existing approaches aren't scalable for large datasets due to training efficiency constraints. Thus, LLMs only see a small fraction of the datasets (e.g., less than 10%) instead of the whole datasets, limiting their exposure to the full training space. To address these problems, we propose RecLoRA. This model incorporates a Personalized LoRA module that maintains independent LoRAs for different users and a Long-Short Modality Retriever that retrieves different history lengths for different modalities, significantly improving performance while adding minimal time cost. Furthermore, we design a Few2Many Learning Strategy, using a conventional recommendation model as a lens to magnify small training spaces to full spaces. Extensive experiments on public datasets demonstrate the efficacy of our RecLoRA compared to existing baseline models.

Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendation

TL;DR

RecLoRA tackles lifelong personalized LLM-based recommendation by introducing a per-user, parameter-efficient adaptation mechanism (Personalized LoRA) coupled with a soft-routing gate, a Long-Short Modality Retriever to balance history length across modalities, and a Few2Many magnifier that leverages a full-space conventional recommender to guide few-shot tuning of the LLM. The approach enables per-user LoRA customization, broader exposure to the training space without expanding fine-tuning cost, and efficient lifelong user modeling. Empirical results on three public datasets show RecLoRA outperforms strong baselines and demonstrates notable gains in few-shot settings, with ablations confirming the importance of all components. The work advances practical deployment of LLM-based recommender systems by addressing personalization, sequence length, and scalability together, with potential implications for real-world large-scale recommendation tasks.

Abstract

We primarily focus on the field of large language models (LLMs) for recommendation, which has been actively explored recently and poses a significant challenge in effectively enhancing recommender systems with logical reasoning abilities and open-world knowledge. Current mainstream efforts mainly center around injecting personalized information from recommendation models into LLMs by customizing input templates or aligning representations between semantic and recommendation spaces at the prediction layer. However, they face three significant limitations: (1) LoRA is mostly used as a core component in existing works, but personalization is not well established in LoRA parameters as the LoRA matrix shared by every user may not cater to different users' characteristics, leading to suboptimal performance. (2) Although lifelong personalized behavior sequences are ideal for personalization, their use raises effectiveness and efficiency issues since LLMs require escalating training and inference time to extend text lengths. (3) Existing approaches aren't scalable for large datasets due to training efficiency constraints. Thus, LLMs only see a small fraction of the datasets (e.g., less than 10%) instead of the whole datasets, limiting their exposure to the full training space. To address these problems, we propose RecLoRA. This model incorporates a Personalized LoRA module that maintains independent LoRAs for different users and a Long-Short Modality Retriever that retrieves different history lengths for different modalities, significantly improving performance while adding minimal time cost. Furthermore, we design a Few2Many Learning Strategy, using a conventional recommendation model as a lens to magnify small training spaces to full spaces. Extensive experiments on public datasets demonstrate the efficacy of our RecLoRA compared to existing baseline models.
Paper Structure (31 sections, 16 equations, 6 figures, 4 tables)

This paper contains 31 sections, 16 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Problem Formulation
  4. Conventional Recommendation Models
  5. Large Language Models as Recommenders
  6. Low-Rank Adaptation of LLMs
  7. Methodology
  8. Overview of RecLoRA
  9. Personalized Low-Rank Adaption
  10. Few2Many Magnifier Strategy
  11. Long-Short Modality Retriever
  12. Experiments
  13. Experiment Setup
  14. Datasets
  15. Evaluation Metrics
  16. ...and 16 more sections

Figures (6)

  • Figure 1: The illustration of lifelong behavior sequence and personalized low-rank adaption (LoRA) of large language models for recommendation.
  • Figure 2: The illustration of textual input-output pair.
  • Figure 3: The model structure and learning strategy of our proposed RecLoRA framework.
  • Figure 4: The AUC performance of RecLoRA w.r.t. different meta-LoRA numbers $N_m$. RecLoRA manages to have already performed very well starting from $N_m=2$.
  • Figure 5: The performance of RecLoRA w.r.t. different history length in ID inputs.
  • ...and 1 more figures