Multifaceted User Modeling in Recommendation: A Federated Foundation Models Approach

TL;DR

MRFF introduces a privacy-preserving, federated foundation-model recommender that trains lightweight, client-side models from scratch and employs a group gating network to allocate users to private user-specific FFNs and shared group-level FFNs. This multifaceted user modeling balances personalization and group-level learning while reducing deployment and communication overhead, enabling scalable recommendations on resource-constrained devices. Empirical results across multiple datasets and backbones show consistent improvements in CTR prediction, with analyses validating the balance loss mechanism and practical feasibility including efficiency and privacy safeguards. The work demonstrates a viable path to privacy-aware, personalized recommendations using distributed, foundation-model–based learning.

Abstract

Multifaceted user modeling aims to uncover fine-grained patterns and learn representations from user data, revealing their diverse interests and characteristics, such as profile, preference, and personality. Recent studies on foundation model-based recommendation have emphasized the Transformer architecture's remarkable ability to capture complex, non-linear user-item interaction relationships. This paper aims to advance foundation model-based recommendersystems by introducing enhancements to multifaceted user modeling capabilities. We propose a novel Transformer layer designed specifically for recommendation, using the self-attention mechanism to capture sequential user-item interaction patterns. Specifically, we design a group gating network to identify user groups, enabling hierarchical discovery across different layers, thereby capturing the multifaceted nature of user interests through multiple Transformer layers. Furthermore, to broaden the data scope and further enhance multifaceted user modeling, we extend the framework to a federated setting, enabling the use of private datasets while ensuring privacy. Experimental validations on benchmark datasets demonstrate the superior performance of our proposed method. Code is available.

Figures (4)

• Figure 1: The framework of MRFF. The right side illustrates the workflow of our method. Each client trains a lightweight foundation model using personal data. In the transformer layers of the local model, we propose a multifaceted user modeling mechanism to enhance user personalization, with details summarized on the left side. Specifically, we introduce a group gating network after the attention module to direct users to specific FFNs, alongside the user-specific FFN, for forward propagation. During iterative optimization between the server and clients, clients maintain user embeddings and user-specific FFN as private modules to learn user-level personalization. For other parameters, the server aggregates them either globally or by group.
• Figure 2: Efficacy analysis of balance loss. The horizontal axis denotes the federated optimization rounds, and the vertical axis shows the number of users. The upper and lower subfigures display user grouping results for model’s two transformer blocks.
• Figure 3: Impact of coefficient of the balance loss on model performace.
• Figure 4: Impact of the number of user groups on model performance.