Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

X-Cross: Dynamic Integration of Language Models for Cross-Domain Sequential Recommendation

Guy Hadad, Haggai Roitman, Yotam Eshel, Bracha Shapira, Lior Rokach

TL;DR

X-Cross tackles the challenge of adapting sequential recommender systems to new domains with limited labeled data by dynamically integrating multiple source-domain language-model encoders that were fine-tuned with LoRA adapters. The method operates layer-by-layer to refine domain-specific representations and then aggregates them to score candidate items, achieving performance close to or surpassing LoRA-fine-tuned baselines while using only a fraction of the parameters. Empirically on Amazon-domain datasets, X-Cross demonstrates data-efficient cross-domain transfer (requiring 50-75% less target-domain data) and outperforms other cross-domain baselines, highlighting its scalability and robustness in data-constrained environments. Overall, the approach offers a practical, parameter-efficient pathway for rapid cross-domain adaptation in recommender systems.

Abstract

As new products are emerging daily, recommendation systems are required to quickly adapt to possible new domains without needing extensive retraining. This work presents ``X-Cross'' -- a novel cross-domain sequential-recommendation model that recommends products in new domains by integrating several domain-specific language models; each model is fine-tuned with low-rank adapters (LoRA). Given a recommendation prompt, operating layer by layer, X-Cross dynamically refines the representation of each source language model by integrating knowledge from all other models. These refined representations are propagated from one layer to the next, leveraging the activations from each domain adapter to ensure domain-specific nuances are preserved while enabling adaptability across domains. Using Amazon datasets for sequential recommendation, X-Cross achieves performance comparable to a model that is fine-tuned with LoRA, while using only 25% of the additional parameters. In cross-domain tasks, such as adapting from Toys domain to Tools, Electronics or Sports, X-Cross demonstrates robust performance, while requiring about 50%-75% less fine-tuning data than LoRA to make fine-tuning effective. Furthermore, X-Cross achieves significant improvement in accuracy over alternative cross-domain baselines. Overall, X-Cross enables scalable and adaptive cross-domain recommendations, reducing computational overhead and providing an efficient solution for data-constrained environments.

X-Cross: Dynamic Integration of Language Models for Cross-Domain Sequential Recommendation

TL;DR

X-Cross tackles the challenge of adapting sequential recommender systems to new domains with limited labeled data by dynamically integrating multiple source-domain language-model encoders that were fine-tuned with LoRA adapters. The method operates layer-by-layer to refine domain-specific representations and then aggregates them to score candidate items, achieving performance close to or surpassing LoRA-fine-tuned baselines while using only a fraction of the parameters. Empirically on Amazon-domain datasets, X-Cross demonstrates data-efficient cross-domain transfer (requiring 50-75% less target-domain data) and outperforms other cross-domain baselines, highlighting its scalability and robustness in data-constrained environments. Overall, the approach offers a practical, parameter-efficient pathway for rapid cross-domain adaptation in recommender systems.

Abstract

As new products are emerging daily, recommendation systems are required to quickly adapt to possible new domains without needing extensive retraining. This work presents ``X-Cross'' -- a novel cross-domain sequential-recommendation model that recommends products in new domains by integrating several domain-specific language models; each model is fine-tuned with low-rank adapters (LoRA). Given a recommendation prompt, operating layer by layer, X-Cross dynamically refines the representation of each source language model by integrating knowledge from all other models. These refined representations are propagated from one layer to the next, leveraging the activations from each domain adapter to ensure domain-specific nuances are preserved while enabling adaptability across domains. Using Amazon datasets for sequential recommendation, X-Cross achieves performance comparable to a model that is fine-tuned with LoRA, while using only 25% of the additional parameters. In cross-domain tasks, such as adapting from Toys domain to Tools, Electronics or Sports, X-Cross demonstrates robust performance, while requiring about 50%-75% less fine-tuning data than LoRA to make fine-tuning effective. Furthermore, X-Cross achieves significant improvement in accuracy over alternative cross-domain baselines. Overall, X-Cross enables scalable and adaptive cross-domain recommendations, reducing computational overhead and providing an efficient solution for data-constrained environments.
Paper Structure (29 sections, 10 equations, 4 figures, 4 tables)

This paper contains 29 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Models for Sequential-Recommendation
  4. Cross-Domain Recommendation
  5. Mixtures of LoRAs
  6. Recommendation Framework
  7. Background
  8. Cross-Domain Sequential Recommendation Task
  9. Sequential Recommendation as a Multiple-Choice Problem
  10. Model Training
  11. LoRA for Domain Specific Fine-Tuning
  12. The X-Cross Model
  13. Stage 1: Concatenation of source domain representations
  14. Stage 2: Dynamic scaling
  15. Stage 3: Representation Refinement and Integration:
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Example prompt for sequential recommendation task having a user history with 5 items and a candidate item.
  • Figure 2: X-Cross model architecture. Each source domain language model is implemented with several Transformer (vertical) layers. On the left side: at each layer, the "hot"-trainable integrator receives activations from the "frozen" layers and then passes the integrated representations to the next layer. On the right side: a "zoom-in" into an X-Cross integrator located at one of the network layers.
  • Figure 3: Accuracy (Hit@1) comparison across datasets for X-Cross and LoRA. The dashed red-line denotes the performance of the reference model.
  • Figure 4: Accuracy (Hit@1) vs number of layers.