Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training

TL;DR

This paper tackles data sparsity and cross-domain generalization in recommender systems by introducing CPRec, an all-domain continual pre-training framework that aligns LLMs' open-world semantic knowledge with domain-specific collaborative patterns. It builds a unified, domain-agnostic CPT pipeline that structures multi-domain user behaviors into domain-specific and all-domain mixed sequences, uses a purely textual unified prompt, and employs a Warmup-Stable-Annealing learning rate scheduler to smoothly transfer knowledge. Extensive experiments across seven CPT domains and five unseen downstream datasets show CPRec achieving state-of-the-art performance, strong robustness across domains, and meaningful zero-shot generalization, even with limited SFT on target domains. The approach offers a scalable, efficient path for deploying LLM-based recommendations in diverse platforms, bridging semantic understanding and collaborative filtering, with potential for further improvements via more data sources and DPO-based post-training.

Abstract

Recent research efforts have investigated how to integrate Large Language Models (LLMs) into recommendation, capitalizing on their semantic comprehension and open-world knowledge for user behavior understanding. These approaches predominantly employ supervised fine-tuning on single-domain user interactions to adapt LLMs for specific recommendation tasks. However, they typically encounter dual challenges: the mismatch between general language representations and domain-specific preference patterns, as well as the limited adaptability to multi-domain recommendation scenarios. To bridge these gaps, we introduce CPRec -- an All-domain Continual Pre-Training framework for Recommendation -- designed to holistically align LLMs with universal user behaviors through the continual pre-training paradigm. Specifically, we first design a unified prompt template and organize users' multi-domain behaviors into domain-specific behavioral sequences and all-domain mixed behavioral sequences that emulate real-world user decision logic. To optimize behavioral knowledge infusion, we devise a Warmup-Stable-Annealing learning rate schedule tailored for the continual pre-training paradigm in recommendation to progressively enhance the LLM's capability in knowledge adaptation from open-world knowledge to universal recommendation tasks. To evaluate the effectiveness of our CPRec, we implement it on a large-scale dataset covering seven domains and conduct extensive experiments on five real-world datasets from two distinct platforms. Experimental results confirm that our continual pre-training paradigm significantly mitigates the semantic-behavioral discrepancy and achieves state-of-the-art performance in all recommendation scenarios. The source code will be released upon acceptance.

Figures (5)

• Figure 1: The modality adaptation pathway in our work. It progressively shifts the open-world knowledge from the universal semantic modality to the specific collaborative modality via the general user behavioral patterns embedded in our all-domain continual pre-training paradigm.
• Figure 2: Results of SASRec, LLaRA w/o SFT, CD-LLaRA w/o SFT, CD-LLaRA, and LLaRA on two datasets. Removing the tailored SFT process results in a cliff-like performance drop, implying the notable gap between open-world semantic knowledge in LLM and collaborative information in RS. A simple cross-domain pre-trained LLM can alleviate it while there is still room for further improvement.
• Figure 3: The overall structure of CPRec, which serves as the bridge between the pre-training of LLM and the SFT on a specific domain. It incorporates multiple single-domain behavioral data and all-domain mixed data into training via WSA scheduler for seamless modality adaptation.
• Figure 4: Results of CPRec and its ablation versions on datasets from two diverse platforms. All components are effective.
• Figure 5: Impact of diverse training data volume on downstream datasets. Generally, more pre-training data is effective.