Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fisher-Informed Parameterwise Aggregation for Federated Learning with Heterogeneous Data

Zhipeng Chang, Ting He, Wenrui Hao

TL;DR

This work addresses the challenge of non-IID data in federated learning by moving beyond uniform client-level scaling to parameterwise aggregation guided by the Fisher Information Matrix. FIPA uses a low-rank spectral approximation of each client's FIM to compute parameter-specific weights, enabling direction-aware updates that better reflect data informativeness. The method is paired with a two-stage training protocol (warmup followed by FIPA refinement) and supports parameter-efficient fine-tuning for large models, maintaining practicality through subspace-based computation and QR-based server merges. Empirically, FIPA improves accuracy across nonlinear function fitting, PDE learning with PINNs, and image classification under varying heterogeneity, and it can complement existing client-side optimization strategies. The results suggest that leveraging Fisher-guided, parameterwise aggregation yields robust, scalable gains in heterogeneous federated settings.

Abstract

Federated learning aggregates model updates from distributed clients, but standard first order methods such as FedAvg apply the same scalar weight to all parameters from each client. Under non-IID data, these uniformly weighted updates can be strongly misaligned across clients, causing client drift and degrading the global model. Here we propose Fisher-Informed Parameterwise Aggregation (FIPA), a second-order aggregation method that replaces client-level scalar weights with parameter-specific Fisher Information Matrix (FIM) weights, enabling true parameter-level scaling that captures how each client's data uniquely influences different parameters. With low-rank approximation, FIPA remains communication- and computation-efficient. Across nonlinear function regression, PDE learning, and image classification, FIPA consistently improves over averaging-based aggregation, and can be effectively combined with state-of-the-art client-side optimization algorithms to further improve image classification accuracy. These results highlight the benefits of FIPA for federated learning under heterogeneous data distributions.

Fisher-Informed Parameterwise Aggregation for Federated Learning with Heterogeneous Data

TL;DR

This work addresses the challenge of non-IID data in federated learning by moving beyond uniform client-level scaling to parameterwise aggregation guided by the Fisher Information Matrix. FIPA uses a low-rank spectral approximation of each client's FIM to compute parameter-specific weights, enabling direction-aware updates that better reflect data informativeness. The method is paired with a two-stage training protocol (warmup followed by FIPA refinement) and supports parameter-efficient fine-tuning for large models, maintaining practicality through subspace-based computation and QR-based server merges. Empirically, FIPA improves accuracy across nonlinear function fitting, PDE learning with PINNs, and image classification under varying heterogeneity, and it can complement existing client-side optimization strategies. The results suggest that leveraging Fisher-guided, parameterwise aggregation yields robust, scalable gains in heterogeneous federated settings.

Abstract

Federated learning aggregates model updates from distributed clients, but standard first order methods such as FedAvg apply the same scalar weight to all parameters from each client. Under non-IID data, these uniformly weighted updates can be strongly misaligned across clients, causing client drift and degrading the global model. Here we propose Fisher-Informed Parameterwise Aggregation (FIPA), a second-order aggregation method that replaces client-level scalar weights with parameter-specific Fisher Information Matrix (FIM) weights, enabling true parameter-level scaling that captures how each client's data uniquely influences different parameters. With low-rank approximation, FIPA remains communication- and computation-efficient. Across nonlinear function regression, PDE learning, and image classification, FIPA consistently improves over averaging-based aggregation, and can be effectively combined with state-of-the-art client-side optimization algorithms to further improve image classification accuracy. These results highlight the benefits of FIPA for federated learning under heterogeneous data distributions.
Paper Structure (23 sections, 1 theorem, 41 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 23 sections, 1 theorem, 41 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Results
  3. FIPA Overview and Workflow
  4. Function Fitting
  5. 1D function fitting.
  6. 2D function fitting.
  7. PDE Solving with PINNs
  8. 1D nonlinear elliptic equations.
  9. $d$-dimensional Poisson equations.
  10. Image Classification on Public Datasets
  11. Discussion
  12. Methods
  13. Problem setup and notation
  14. Fisher information matrix.
  15. FIPA aggregation and low-rank approximation
  16. ...and 8 more sections

Key Result

Theorem 1

Under the zero-residual regular regime (there exists a solution $\bm{\theta}^\star$ with $\bm{\xi}(\bm{\theta}^\star)=\mathbf{0}$, the Jacobian $\mathbf{J}$ is Lipschitz continuous near $\bm{\theta}^\star$, and $\mathbf{J}(\bm{\theta}^\star)$ has full column rank), with fixed damping parameter $\gam where $\rho\in(0,1)$ is the contraction rate, and $C$ is a constant such that $\delta^{(k)}\le C/(k

Figures (4)

  • Figure 1: Overview of the FIPA workflow and aggregation mechanism. The server aggregates client updates using parameterwise matrix weights $\mathbf{B}_m^{(k)}$\ref{['eq:bm']}. The parameter-server federated learning architecture illustrates the communication flow; heterogeneous client data are shown in different colors.
  • Figure 2: Function fitting results. (A) 1D function fitting: test loss vs. time (top) and fitted function quality (bottom) for target $u(x) = \sin(n\pi x)$ with different client partitions. (B) Adaptive method performance: test loss evolution (left), adaptive local iterations $\tau_m^{(k)}$ (middle), and adaptive rank $r$ (right) for the 2-client $\sin(2\pi x)$ case. (C) Comparison with centralized GN: test loss vs. communication rounds. The black dashed line shows the approximate slope of centralized GN convergence. (D--F) 2D Gaussian-mixture target: test loss vs. time (D), target function and client distribution (E), and pointwise error map (F).
  • Figure 3: PDE solving results. (A) Relative $L^2$ error vs. time for different 1D nonlinear elliptic PDE problems, from left to right: Allen-Cahn, Bratu, Fisher, Reaction-Diffusion. (B) Solution quality for 1D nonlinear elliptic PDEs. (C--E)$d$-dimensional Poisson equations ($d \in \{1, 2, 3, 5\}$) with two-client split: relative $L^2$ error vs. time.
  • Figure 4: Impact of different warmup rounds on FIPA performance. (A) Test accuracy vs. communication rounds for ResNet-20 under $\alpha=0.05$ when FIPA is initiated after different warmup round counts. (B) Maximum improvement of FIPA over FedAvg within the first 15 rounds after warmup across different network architectures (CNN-8k, CNN-23k, CNN-207k, ResNet-20) and heterogeneity levels ($\alpha \in \{0.01, 0.05\}$).

Theorems & Definitions (1)

  • Theorem 1: End-to-end bound via a centralized damped GN reference