Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Adaptation for Foundation Model-based Recommendations

Chunxu Zhang, Guodong Long, Hongkuan Guo, Xiao Fang, Yang Song, Zhaojie Liu, Guorui Zhou, Zijian Zhang, Yang Liu, Bo Yang

TL;DR

This work addresses the challenge of adapting foundation-model-based recommender systems under privacy and communication constraints. It introduces FedPA, a federated framework that leverages a pre-trained foundation model backbone and lightweight, two-level low-rank adapters (user-level and user-group-level) paired with an adaptive gate to fuse common knowledge with personalized signals, while updating only personalization parameters. The approach is complemented by knowledge distillation to enable edge-device deployment and Local Differential Privacy to strengthen privacy protection, with extensive experiments on four KuaiShou-derived datasets showing superior performance and reduced communication costs. The results indicate FedPA’s practical potential for privacy-preserving, scalable, foundation-model-based recommendations in real-world settings, especially when on-device computation is limited and data remains private.

Abstract

With the recent success of large language models, particularly foundation models with generalization abilities, applying foundation models for recommendations becomes a new paradigm to improve existing recommendation systems. It becomes a new open challenge to enable the foundation model to capture user preference changes in a timely manner with reasonable communication and computation costs while preserving privacy. This paper proposes a novel federated adaptation mechanism to enhance the foundation model-based recommendation system in a privacy-preserving manner. Specifically, each client will learn a lightweight personalized adapter using its private data. The adapter then collaborates with pre-trained foundation models to provide recommendation service efficiently with fine-grained manners. Importantly, users' private behavioral data remains secure as it is not shared with the server. This data localization-based privacy preservation is embodied via the federated learning framework. The model can ensure that shared knowledge is incorporated into all adapters while simultaneously preserving each user's personal preferences. Experimental results on four benchmark datasets demonstrate our method's superior performance. Implementation code is available to ease reproducibility.

Federated Adaptation for Foundation Model-based Recommendations

TL;DR

This work addresses the challenge of adapting foundation-model-based recommender systems under privacy and communication constraints. It introduces FedPA, a federated framework that leverages a pre-trained foundation model backbone and lightweight, two-level low-rank adapters (user-level and user-group-level) paired with an adaptive gate to fuse common knowledge with personalized signals, while updating only personalization parameters. The approach is complemented by knowledge distillation to enable edge-device deployment and Local Differential Privacy to strengthen privacy protection, with extensive experiments on four KuaiShou-derived datasets showing superior performance and reduced communication costs. The results indicate FedPA’s practical potential for privacy-preserving, scalable, foundation-model-based recommendations in real-world settings, especially when on-device computation is limited and data remains private.

Abstract

With the recent success of large language models, particularly foundation models with generalization abilities, applying foundation models for recommendations becomes a new paradigm to improve existing recommendation systems. It becomes a new open challenge to enable the foundation model to capture user preference changes in a timely manner with reasonable communication and computation costs while preserving privacy. This paper proposes a novel federated adaptation mechanism to enhance the foundation model-based recommendation system in a privacy-preserving manner. Specifically, each client will learn a lightweight personalized adapter using its private data. The adapter then collaborates with pre-trained foundation models to provide recommendation service efficiently with fine-grained manners. Importantly, users' private behavioral data remains secure as it is not shared with the server. This data localization-based privacy preservation is embodied via the federated learning framework. The model can ensure that shared knowledge is incorporated into all adapters while simultaneously preserving each user's personal preferences. Experimental results on four benchmark datasets demonstrate our method's superior performance. Implementation code is available to ease reproducibility.
Paper Structure (25 sections, 9 equations, 1 figure, 5 tables)

This paper contains 25 sections, 9 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundation Models for Recommendation
  4. Federated Recommendation System
  5. Methodology
  6. Framework Overview
  7. Base Model
  8. Personalized Adapter
  9. Adaptive Gate Learning Mechanism
  10. Optimization and Model Scalability
  11. Optimization Objective
  12. Efficient Parameter Update
  13. Discussion about Model Scalability
  14. Privacy-Preserving Enhanced FedPA
  15. Experiment
  16. ...and 10 more sections

Figures (1)

  • Figure 1: The framework of FedPA. The left part represents the workflow of our method. Each client learns the local recommendation model based on personal data, initializing it with parameters from the pre-trained model. During training, we only update the parameters about user personalization modeling and keep others frozen. The server is responsible for globally aggregating the shared parameters to transmit common information among clients. The right part illustrates the details of employing the adaptive gate learning mechanism to fuse the common information from each layer of the prediction function and user personalization from the personalized adapter at two granularities. It is worth noting that the user-level adapter is a private module and the user-group-level adapter is a public module.