Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sharpness-Aware Minimization for Generalized Embedding Learning in Federated Recommendation

Fengyuan Yu, Xiaohua Feng, Yuyuan Li, Changwang Zhang, Jun Wang, Chaochao Chen

Abstract

Federated recommender systems enable collaborative model training while keeping user interaction data local and sharing only essential model parameters, thereby mitigating privacy risks. However, existing methods overlook a critical issue, i.e., the stable learning of a generalized item embedding throughout the federated recommender system training process. Item embedding plays a central role in facilitating knowledge sharing across clients. Yet, under the cross-device setting, local data distributions exhibit significant heterogeneity and sparsity, exacerbating the difficulty of learning generalized embeddings. These factors make the stable learning of generalized item embeddings both indispensable for effective federated recommendation and inherently difficult to achieve. To fill this gap, we propose a new federated recommendation framework, named Federated Recommendation with Generalized Embedding Learning (FedRecGEL). We reformulate the federated recommendation problem from an item-centered perspective and cast it as a multi-task learning problem, aiming to learn generalized embeddings throughout the training procedure. Based on theoretical analysis, we employ sharpness-aware minimization to address the generalization problem, thereby stabilizing the training process and enhancing recommendation performance. Extensive experiments on four datasets demonstrate the effectiveness of FedRecGEL in significantly improving federated recommendation performance. Our code is available at https://github.com/anonymifish/FedRecGEL.

Sharpness-Aware Minimization for Generalized Embedding Learning in Federated Recommendation

Abstract

Federated recommender systems enable collaborative model training while keeping user interaction data local and sharing only essential model parameters, thereby mitigating privacy risks. However, existing methods overlook a critical issue, i.e., the stable learning of a generalized item embedding throughout the federated recommender system training process. Item embedding plays a central role in facilitating knowledge sharing across clients. Yet, under the cross-device setting, local data distributions exhibit significant heterogeneity and sparsity, exacerbating the difficulty of learning generalized embeddings. These factors make the stable learning of generalized item embeddings both indispensable for effective federated recommendation and inherently difficult to achieve. To fill this gap, we propose a new federated recommendation framework, named Federated Recommendation with Generalized Embedding Learning (FedRecGEL). We reformulate the federated recommendation problem from an item-centered perspective and cast it as a multi-task learning problem, aiming to learn generalized embeddings throughout the training procedure. Based on theoretical analysis, we employ sharpness-aware minimization to address the generalization problem, thereby stabilizing the training process and enhancing recommendation performance. Extensive experiments on four datasets demonstrate the effectiveness of FedRecGEL in significantly improving federated recommendation performance. Our code is available at https://github.com/anonymifish/FedRecGEL.
Paper Structure (33 sections, 2 theorems, 19 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 33 sections, 2 theorems, 19 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Recommendation
  4. Sharpness Aware Minimization
  5. Preliminaries
  6. Problem Statement
  7. Problem Reformulating in Item-Centered Perspective
  8. Methodology
  9. Theoretical Analysis
  10. Relating the General Loss to the Empirical Loss
  11. From Empirical Loss Minimization to SAM
  12. Computational Implementation
  13. Update the non-shared parts
  14. Update the shared part
  15. Implementation
  16. ...and 18 more sections

Key Result

lemma 1

With the assumption that adding a Gaussian perturbation will raise the test error: $\mathcal{L}_{\mathcal{D}}(\boldsymbol{\theta}^k) \le \mathbb{E}_{\boldsymbol{\varepsilon}\sim\mathcal{N}(0,\sigma^2 \mathbb{I})}\bigl[\mathcal{L}_{\mathcal{D}}(\boldsymbol{\theta}^k+\boldsymbol{\varepsilon})\bigr]$. where $f^k\left(\|\boldsymbol{\theta}^k\|_2^2\right)$ is a regularization term, equals to and $L$

Figures (3)

  • Figure 1: 3D surface plots of the post-convergence landscapes for models trained with FedRecGEL and with FedNCF.
  • Figure 2: Effect of hyperparameter $\rho_{\mathrm{ur}}$ and $\rho_{\mathrm{co}}$. We conduct experiments on all four datasets. We report HR@10 and NDCG@10 to represent the recommendation performance.
  • Figure 3: The contour map visualization of the post-convergence landscapes for models trained with FedRecGEL and with FedNCF across all four datasets.

Theorems & Definitions (2)

  • lemma 1: Multi Gaussian-Perturbed PAC Bound
  • lemma 2: Hierarchical SAM