Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data

Feifei Wang, Huiyun Tang, Yang Li

TL;DR

This work tackles data heterogeneity in federated learning by decomposing neural network elements into shared and client-specific components (FedSplit), enabling concurrent global sharing and local personalization. It formalizes the FedSplit objective, provides NTK-based linear convergence guarantees and generalization bounds, and introduces FedFac—an operationalization using factor analysis to perform the decomposition. Through simulations and real-data experiments (FEMNIST, Shakespeare, CIFAR10/100), FedFac consistently outperforms strong FL baselines, with dynamic FedFac offering the clearest advantages and improved cross-client fairness. The approach yields practical benefits for personalized deployment in heterogeneous FL settings, while highlighting computational trade-offs and future work on automatic decomposition and privacy protections.

Abstract

Federated learning is an emerging distributed machine learning framework aiming at protecting data privacy. Data heterogeneity is one of the core challenges in federated learning, which could severely degrade the convergence rate and prediction performance of deep neural networks. To address this issue, we develop a novel personalized federated learning framework for heterogeneous data, which we refer to as FedSplit. This modeling framework is motivated by the finding that, data in different clients contain both common knowledge and personalized knowledge. Then the hidden elements in each neural layer can be split into the shared and personalized groups. With this decomposition, a novel objective function is established and optimized. We demonstrate FedSplit enjoyers a faster convergence speed than the standard federated learning method both theoretically and empirically. The generalization bound of the FedSplit method is also studied. To practically implement the proposed method on real datasets, factor analysis is introduced to facilitate the decoupling of hidden elements. This leads to a practically implemented model for FedSplit and we further refer to as FedFac. We demonstrated by simulation studies that, using factor analysis can well recover the underlying shared/personalized decomposition. The superior prediction performance of FedFac is further verified empirically by comparison with various state-of-the-art federated learning methods on several real datasets.

Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data

TL;DR

This work tackles data heterogeneity in federated learning by decomposing neural network elements into shared and client-specific components (FedSplit), enabling concurrent global sharing and local personalization. It formalizes the FedSplit objective, provides NTK-based linear convergence guarantees and generalization bounds, and introduces FedFac—an operationalization using factor analysis to perform the decomposition. Through simulations and real-data experiments (FEMNIST, Shakespeare, CIFAR10/100), FedFac consistently outperforms strong FL baselines, with dynamic FedFac offering the clearest advantages and improved cross-client fairness. The approach yields practical benefits for personalized deployment in heterogeneous FL settings, while highlighting computational trade-offs and future work on automatic decomposition and privacy protections.

Abstract

Federated learning is an emerging distributed machine learning framework aiming at protecting data privacy. Data heterogeneity is one of the core challenges in federated learning, which could severely degrade the convergence rate and prediction performance of deep neural networks. To address this issue, we develop a novel personalized federated learning framework for heterogeneous data, which we refer to as FedSplit. This modeling framework is motivated by the finding that, data in different clients contain both common knowledge and personalized knowledge. Then the hidden elements in each neural layer can be split into the shared and personalized groups. With this decomposition, a novel objective function is established and optimized. We demonstrate FedSplit enjoyers a faster convergence speed than the standard federated learning method both theoretically and empirically. The generalization bound of the FedSplit method is also studied. To practically implement the proposed method on real datasets, factor analysis is introduced to facilitate the decoupling of hidden elements. This leads to a practically implemented model for FedSplit and we further refer to as FedFac. We demonstrated by simulation studies that, using factor analysis can well recover the underlying shared/personalized decomposition. The superior prediction performance of FedFac is further verified empirically by comparison with various state-of-the-art federated learning methods on several real datasets.
Paper Structure (27 sections, 2 theorems, 21 equations, 10 figures, 8 tables, 2 algorithms)

This paper contains 27 sections, 2 theorems, 21 equations, 10 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Heterogeneity adjusted global models
  4. Personalization models
  5. Split-personalization models
  6. The FedSplit Methodology
  7. Problem formulation
  8. Model estimation and prediction
  9. Theoretical properties
  10. Factor-Assisted Decomposition
  11. Decomposition with factor analysis
  12. Static and dynamic algorithms
  13. Effectiveness of using factor analysis
  14. Simulation Setup
  15. Simulation results
  16. ...and 12 more sections

Key Result

Theorem 1

Suppose Assumption assum holds. Let $\lambda^s = \lambda_{min}\left(\mathbf{H}^{s\infty}\right)$, $\gamma^s = \lambda_{max}\left(\mathbf{H}^{s\infty}\right)/\lambda_{min}\left(\mathbf{H}^{s\infty}\right)$, $\lambda^p = \lambda_{min}\left(\mathbf{H}^{p\infty}\right)$, $\gamma^p = \lambda_{max}\left(\

Figures (10)

  • Figure 1: The heatmaps of outputs generated by each neuron in each client in the IID and non-IID cases.
  • Figure 2: The histograms of entropy of outputs generated by each neuron across all clients in the IID and non-IID cases.
  • Figure 3: The illustration of neurons decomposition in DNNs under the FedSplit framework. The light blue circles represent the shared elements, which are updated by all clients, while circles in other colors represent the client-specific elements, which are only updated locally.
  • Figure 4: Data preparation for factor analysis.
  • Figure 5: The overall framework of FedFac.
  • ...and 5 more figures

Theorems & Definitions (4)

  • Definition 1: Neural tangent kernel
  • Definition 2: Gram matrix
  • Theorem 1: Convergence Rate
  • Theorem 2: Generalization Bounds