Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation

TL;DR

The paper tackles the problem that large language models struggle to extract meaningful signals from long user behavior sequences in recommendation. It introduces ReLLaX, a full-stack framework combining semantic retrieval of user histories (SUBR), soft prompt augmentation (SPA), and fully interactive LoRA (CFLoRA) to enable robust lifelong sequence understanding. The authors provide a theoretical lens comparing CFLoRA to existing LoRA-based LLM4Rec methods and demonstrate substantial gains on three public CTR datasets, along with extensive ablations and case studies. These contributions offer a path toward more effective and data-efficient LLM-based recommendation, though they acknowledge slower inference and propose future efficiency-oriented directions.

Abstract

In this paper, we address the lifelong sequential behavior incomprehension problem in large language models (LLMs) for recommendation, where LLMs struggle to extract useful information from long user behavior sequences, even within their context limits. To tackle this, we propose ReLLaX (Retrieval-enhanced Large Language models Plus), a framework offering optimization across data, prompt, and parameter levels. At the data level, we introduce Semantic User Behavior Retrieval (SUBR) to reduce sequence heterogeneity, making it easier for LLMs to extract key information. For prompt-level enhancement, we employ Soft Prompt Augmentation (SPA) to inject collaborative knowledge, aligning item representations with recommendation tasks and improving LLMs's exploration of item relationships. Finally, at the parameter level, we propose Component Fully-interactive LoRA (CFLoRA), which enhances LoRA's expressiveness by enabling interactions between its components, allowing better capture of sequential information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e. from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments on three public datasets demonstrate ReLLaX's superiority over existing baselines and its ability to mitigate lifelong sequential behavior incomprehension effectively.

Figures (12)

• Figure 1: The illustration of lifelong sequential behavior incomprehension problem for LLMs. We report the AUC performance of SIM and different LLMs on MovieLens-1M dataset. While SIM enjoys steady performance improvement as the length of behavior sequence $K$ grows, the LLMs generally only peaks at $K=15$ and fails to extract the useful information with further longer sequences (i.e., $K>15$).
• Figure 2: Comprehensive comparison of the ways different LLM4Rec methods employ LoRA for finetuning.
• Figure 3: The overall framework of our proposed ReLLaX, which proposes a full-stack optimization for LLMs to address the lifelong sequence incomprehension in recommendation. The enhancement lies in three different levels, e.g. data, prompt and parameter.
• Figure 4: Illustration of descriptive text for an item (movie).
• Figure 5: Examples of prompt templates for the MovieLens-1M dataset. (a) Vanilla prompt, which directly converts the recommendation data into text. (b) Hard prompt with SUBR, which replaces the most recent $K$ items in (a) with the most semantically relevant $K$ items towards the target item. (c) Comprehensive prompt with SUBR and SPA, extending (b) with the blue tokens, which represent the soft prompt tokens generated by SPA.