MMGRid: Navigating Temporal-aware and Cross-domain Generative Recommendation via Model Merging

TL;DR

This work presents the first systematic study of MM in GR through a contextual lens, and observes that optimal merging weights correlate with context-dependent interaction characteristics, offering practical guidance for weight selection in real-world deployments.

Abstract

Model merging (MM) offers an efficient mechanism for integrating multiple specialized models without access to original training data or costly retraining. While MM has demonstrated success in domains like computer vision, its role in recommender systems (RSs) remains largely unexplored. Recently, Generative Recommendation (GR) has emerged as a new paradigm in RSs, characterized by rapidly growing model scales and substantial computational costs, making MM particularly appealing for cost-sensitive deployment scenarios. In this work, we present the first systematic study of MM in GR through a contextual lens. We focus on a fundamental yet underexplored challenge in real-world: how to merge generative recommenders specialized to different real-world contexts, arising from temporal evolving user behaviors and heterogeneous application domains. To this end, we propose a unified framework MMGRid, a structured contextual grid of GR checkpoints that organizes models trained under diverse contexts induced by temporal evolution and domain diversity. All checkpoints are derived from a shared base LLM but fine-tuned on context-specific data, forming a realistic and controlled model space for systematically analyzing MM across GR paradigms and merging algorithms. Our investigation reveals several key insights. First, training GR models from LLMs can introduce parameter conflicts during merging due to token distribution shifts and objective disparities; such conflicts can be alleviated by disentangling task-aware and context-specific parameter changes via base model replacement. Second, incremental training across contexts induces recency bias, which can be effectively balanced through weighted contextual merging. Notably, we observe that optimal merging weights correlate with context-dependent interaction characteristics, offering practical guidance for weight selection in real-world deployments.

Figures (5)

• Figure 1: Overview of the MMGRid Framework. Generative recommendation (GR) models built on a shared base LLM are organized into a contextual grid across domains and temporal stages. This grid provides a unified model space for studying merging of GR models on contextual scenarios, identifying key challenges and promising directions for solutions.
• Figure 2: Task vector magnitude of fine-tuned model checkpoints with the same base model Qwen3-0.6B.
• Figure 3: Performance heatmap of cross-domain merging.
• Figure 4: Joint performance comparsion of cross-domain merging with different base models.
• Figure 5: Temporal preference shift of fine-tuned models. The vertical dashed line indicates the $\lambda_{temp}$ value that achieves the best performance for each domain and user group.

Theorems & Definitions (5)

• Definition 1: Sequential Recommendation
• Definition 2: Parameter-wise Model Merging
• Definition 3: Cross-Domain Merging
• Definition 4: Temporal Merging
• Definition 5: Temporal Preference Shift Vector