Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continual Low-Rank Adapters for LLM-based Generative Recommender Systems

Hyunsik Yoo, Ting-Wei Li, SeongKu Kang, Zhining Liu, Charlie Xu, Qilin Qi, Hanghang Tong

TL;DR

This work tackles continual learning for LLM-based generative recommender systems, where user preferences drift over time. It introduces PESO, a proximally regularized single evolving LoRA framework that anchors updates to the previous adapter, enabling data-aware, direction-wise guidance in the LoRA subspace. The authors provide a theoretical analysis showing how the proximal term mediates stability and plasticity and instantiate a per-module softmax–KL proximal to preserve module structure. Empirically, PESO outperforms both single evolving LoRA and cumulative LoRA baselines across multiple real-world Amazon datasets, demonstrating a more effective stability–plasticity balance for continual recommendation. The approach offers a scalable, parameter-efficient path to adapt LLM-based recommenders to evolving user behavior with controlled forgetting.

Abstract

While large language models (LLMs) achieve strong performance in recommendation, they face challenges in continual learning as users, items, and user preferences evolve over time. Existing LoRA-based continual methods primarily focus on preserving performance on previous tasks, but this overlooks the unique nature of recommendation: the goal is not to predict past preferences, and outdated preferences can even harm performance when current interests shift significantly. To address this, we propose PESO (Proximally rEgularized Single evolving lOra, a continual adaptation method for LoRA in recommendation. PESO introduces a proximal regularizer that anchors the current adapter to its most recent frozen state, enabling the model to flexibly balance adaptation and preservation, and to better capture recent user behaviors. Theoretically, we show that this proximal design provides data-aware, direction-wise guidance in the LoRA subspace. Empirically, PESO consistently outperforms existing LoRA-based continual learning methods.

Continual Low-Rank Adapters for LLM-based Generative Recommender Systems

TL;DR

This work tackles continual learning for LLM-based generative recommender systems, where user preferences drift over time. It introduces PESO, a proximally regularized single evolving LoRA framework that anchors updates to the previous adapter, enabling data-aware, direction-wise guidance in the LoRA subspace. The authors provide a theoretical analysis showing how the proximal term mediates stability and plasticity and instantiate a per-module softmax–KL proximal to preserve module structure. Empirically, PESO outperforms both single evolving LoRA and cumulative LoRA baselines across multiple real-world Amazon datasets, demonstrating a more effective stability–plasticity balance for continual recommendation. The approach offers a scalable, parameter-efficient path to adapt LLM-based recommenders to evolving user behavior with controlled forgetting.

Abstract

While large language models (LLMs) achieve strong performance in recommendation, they face challenges in continual learning as users, items, and user preferences evolve over time. Existing LoRA-based continual methods primarily focus on preserving performance on previous tasks, but this overlooks the unique nature of recommendation: the goal is not to predict past preferences, and outdated preferences can even harm performance when current interests shift significantly. To address this, we propose PESO (Proximally rEgularized Single evolving lOra, a continual adaptation method for LoRA in recommendation. PESO introduces a proximal regularizer that anchors the current adapter to its most recent frozen state, enabling the model to flexibly balance adaptation and preservation, and to better capture recent user behaviors. Theoretically, we show that this proximal design provides data-aware, direction-wise guidance in the LoRA subspace. Empirically, PESO consistently outperforms existing LoRA-based continual learning methods.

Paper Structure

This paper contains 23 sections, 39 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Analysis of LoRA Variants for Continual Recommendation
  4. Proposed Framework: PESO
  5. Single Evolving LoRA with a Proximal Regularizer
  6. Softmax–KL as a Proximal Regularizer
  7. Experiments
  8. Experimental Settings
  9. Experimental Results and Discussion
  10. Related Works
  11. Conclusion
  12. Conceptual Modeling of Evolving User Preferences
  13. Detailed Theoretical Analysis
  14. Setup and Assumptions
  15. Remark: relinearization per block.
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Conceptual overview of Cumulative LoRA and our proposed PESO with proximal regularizer.
  • Figure 2: Performance comparison of different regularization methods against the previous LoRA.
  • Figure 3: Impact of the scaling weight $\lambda$ for the proximal term on PESO performance.
  • Figure 4: Impact of the learning rate for continual data on model performance.

Theorems & Definitions (2)

  • proof
  • proof