Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TransFR: Transferable Federated Recommendation with Adapter Tuning on Pre-trained Language Models

Honglei Zhang, Zhiwei Li, Haoxuan Li, Xin Zhou, Jie Zhang, Yidong Li

TL;DR

This work tackles cross-domain transferable federated recommendations under privacy constraints by replacing domain-specific item IDs with domain-agnostic textual embeddings derived from pre-trained language models. It introduces TransFR, a framework that trains a compact DistBERT via hierarchical knowledge distillation on public text, then applies federated adapter tuning to tailor the model to local recommender tasks, followed by post-adaptation personalization with a local prediction head. The approach is backed by theoretical analyses of model effectiveness and privacy guarantees, and empirical results on the Amazon dataset show improved transferability to target domains and competitive cold-start performance against strong baselines. The combination of public-text representations, adapter-level federation, and client-specific personalization offers practical benefits for privacy-preserving, scalable cross-domain recommendations.

Abstract

Federated recommendations (FRs), facilitating multiple local clients to collectively learn a global model without disclosing user private data, have emerged as a prevalent on-device service. In conventional FRs, a dominant paradigm is to utilize discrete identities to represent clients and items, which are then mapped to domain-specific embeddings to participate in model training. Despite considerable performance, we reveal three inherent limitations that can not be ignored in federated settings, i.e., non-transferability across domains, ineffectiveness in cold-start settings, and potential privacy violations during federated training. To this end, we propose a transferable federated recommendation model, TransFR, which delicately incorporates the general capabilities empowered by pre-trained models and the personalized abilities by fine-tuning local private data. Specifically, it first learns domain-agnostic representations of items by exploiting pre-trained models with public textual corpora. To tailor for FR tasks, we further introduce efficient federated adapter-tuning and test-time adaptation mechanisms, which facilitate personalized local adapters for each client by fitting their private data distributions. We theoretically prove the advantages of incorporating adapter tuning in FRs regarding both effectiveness and privacy. Through extensive experiments, we show that our TransFR model surpasses several state-of-the-art FRs on transferability.

TransFR: Transferable Federated Recommendation with Adapter Tuning on Pre-trained Language Models

TL;DR

This work tackles cross-domain transferable federated recommendations under privacy constraints by replacing domain-specific item IDs with domain-agnostic textual embeddings derived from pre-trained language models. It introduces TransFR, a framework that trains a compact DistBERT via hierarchical knowledge distillation on public text, then applies federated adapter tuning to tailor the model to local recommender tasks, followed by post-adaptation personalization with a local prediction head. The approach is backed by theoretical analyses of model effectiveness and privacy guarantees, and empirical results on the Amazon dataset show improved transferability to target domains and competitive cold-start performance against strong baselines. The combination of public-text representations, adapter-level federation, and client-specific personalization offers practical benefits for privacy-preserving, scalable cross-domain recommendations.

Abstract

Federated recommendations (FRs), facilitating multiple local clients to collectively learn a global model without disclosing user private data, have emerged as a prevalent on-device service. In conventional FRs, a dominant paradigm is to utilize discrete identities to represent clients and items, which are then mapped to domain-specific embeddings to participate in model training. Despite considerable performance, we reveal three inherent limitations that can not be ignored in federated settings, i.e., non-transferability across domains, ineffectiveness in cold-start settings, and potential privacy violations during federated training. To this end, we propose a transferable federated recommendation model, TransFR, which delicately incorporates the general capabilities empowered by pre-trained models and the personalized abilities by fine-tuning local private data. Specifically, it first learns domain-agnostic representations of items by exploiting pre-trained models with public textual corpora. To tailor for FR tasks, we further introduce efficient federated adapter-tuning and test-time adaptation mechanisms, which facilitate personalized local adapters for each client by fitting their private data distributions. We theoretically prove the advantages of incorporating adapter tuning in FRs regarding both effectiveness and privacy. Through extensive experiments, we show that our TransFR model surpasses several state-of-the-art FRs on transferability.
Paper Structure (23 sections, 3 theorems, 14 equations, 6 figures, 5 tables)

This paper contains 23 sections, 3 theorems, 14 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Recommendation
  4. Transferable Recommendation
  5. The Proposed TransFR Framework
  6. Preliminary
  7. The TransFR Model
  8. Overview
  9. Server-side Execution Procedure.
  10. Client-side Execution Procedure.
  11. Theoretical Analysis
  12. Model Effectiveness.
  13. Privacy Guarantee.
  14. Experiments
  15. Experimental Settings
  16. ...and 8 more sections

Key Result

Theorem 1

Let the global item representation model be a linear transformation $\mathbf{A} \in \mathbb{R}^{L \times f}$ and the local adapters for client $u$ be $\mathbf{B}_u \in \mathbb{R}^{f \times d}$. Then, we have:

Figures (6)

  • Figure 1: Illustration of the differences between standard FR (left) and our TransFR model (right). Existing work operates as a lookup table with domain-specific item embeddings, while TransFR exploits pre-trained language models to produce universal domain-agnostic embeddings.
  • Figure 2: The schematic diagram of TransFR, which includes hierarchical knowledge distillation, federated adapter-tuning and post-adaptation personalization (PAP).
  • Figure 3: Ablation study results towards HR@10 in the three transferable tasks on Amazon dataset.
  • Figure 4: Ablation study results towards NDCG@10 in the three transferable tasks on Amazon dataset.
  • Figure 5: Performance analysis of TransFR with varying numbers of layers in the adapter module.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Theorem 1: Effectiveness of Personalized Adapters
  • Theorem 2: Mutual Information Upper Bound
  • Theorem 3: Mutual Information Bound from DP Mechanism