Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sharpness-aware Federated Graph Learning

Ruiyu Li, Peige Zhao, Guangxia Li, Pengcheng Wu, Xingyu Gao, Zhiqiang Xu

TL;DR

This work tackles the challenge of training graph neural networks on decentralized, privacy-preserving graph data where local distributions are highly non-IID. It introduces SEAL, a Sharpness-aware Federated Graph Learning framework that combines adaptive sharpness-aware minimization with a covariance-based decorrelation regularizer to prevent dimensional collapse and improve cross-client generalization. Across diverse graph classification benchmarks, SEAL consistently outperforms state-of-the-art FGL baselines and shows robust gains under IID, non-IID, cross-dataset, and inter-domain settings, albeit with higher computational cost due to SAM. The work provides thorough ablations clarifying hyperparameter regimes (α, ρ, E) and demonstrates practical impact for federated graph learning in heterogeneous domains.

Abstract

One of many impediments to applying graph neural networks (GNNs) to large-scale real-world graph data is the challenge of centralized training, which requires aggregating data from different organizations, raising privacy concerns. Federated graph learning (FGL) addresses this by enabling collaborative GNN model training without sharing private data. However, a core challenge in FGL systems is the variation in local training data distributions among clients, known as the data heterogeneity problem. Most existing solutions suffer from two problems: (1) The typical optimizer based on empirical risk minimization tends to cause local models to fall into sharp valleys and weakens their generalization to out-of-distribution graph data. (2) The prevalent dimensional collapse in the learned representations of local graph data has an adverse impact on the classification capacity of the GNN model. To this end, we formulate a novel optimization objective that is aware of the sharpness (i.e., the curvature of the loss surface) of local GNN models. By minimizing the loss function and its sharpness simultaneously, we seek out model parameters in a flat region with uniformly low loss values, thus improving the generalization over heterogeneous data. By introducing a regularizer based on the correlation matrix of local representations, we relax the correlations of representations generated by individual local graph samples, so as to alleviate the dimensional collapse of the learned model. The proposed \textbf{S}harpness-aware f\textbf{E}derated gr\textbf{A}ph \textbf{L}earning (SEAL) algorithm can enhance the classification accuracy and generalization ability of local GNN models in federated graph learning. Experimental studies on several graph classification benchmarks show that SEAL consistently outperforms SOTA FGL baselines and provides gains for more participants.

Sharpness-aware Federated Graph Learning

TL;DR

This work tackles the challenge of training graph neural networks on decentralized, privacy-preserving graph data where local distributions are highly non-IID. It introduces SEAL, a Sharpness-aware Federated Graph Learning framework that combines adaptive sharpness-aware minimization with a covariance-based decorrelation regularizer to prevent dimensional collapse and improve cross-client generalization. Across diverse graph classification benchmarks, SEAL consistently outperforms state-of-the-art FGL baselines and shows robust gains under IID, non-IID, cross-dataset, and inter-domain settings, albeit with higher computational cost due to SAM. The work provides thorough ablations clarifying hyperparameter regimes (α, ρ, E) and demonstrates practical impact for federated graph learning in heterogeneous domains.

Abstract

One of many impediments to applying graph neural networks (GNNs) to large-scale real-world graph data is the challenge of centralized training, which requires aggregating data from different organizations, raising privacy concerns. Federated graph learning (FGL) addresses this by enabling collaborative GNN model training without sharing private data. However, a core challenge in FGL systems is the variation in local training data distributions among clients, known as the data heterogeneity problem. Most existing solutions suffer from two problems: (1) The typical optimizer based on empirical risk minimization tends to cause local models to fall into sharp valleys and weakens their generalization to out-of-distribution graph data. (2) The prevalent dimensional collapse in the learned representations of local graph data has an adverse impact on the classification capacity of the GNN model. To this end, we formulate a novel optimization objective that is aware of the sharpness (i.e., the curvature of the loss surface) of local GNN models. By minimizing the loss function and its sharpness simultaneously, we seek out model parameters in a flat region with uniformly low loss values, thus improving the generalization over heterogeneous data. By introducing a regularizer based on the correlation matrix of local representations, we relax the correlations of representations generated by individual local graph samples, so as to alleviate the dimensional collapse of the learned model. The proposed \textbf{S}harpness-aware f\textbf{E}derated gr\textbf{A}ph \textbf{L}earning (SEAL) algorithm can enhance the classification accuracy and generalization ability of local GNN models in federated graph learning. Experimental studies on several graph classification benchmarks show that SEAL consistently outperforms SOTA FGL baselines and provides gains for more participants.

Paper Structure

This paper contains 20 sections, 1 theorem, 14 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Heterogeneity in Federated Learning
  4. Federated Graph Learning
  5. Sharpness-aware Minimization
  6. Preliminaries
  7. Graph Neural Networks
  8. Federated Learning
  9. The SEAL Algorithm
  10. Federated Graph Learning with Sharpness-aware Minimization
  11. Mitigating Dimensional Collapse via Representation Decorrelation
  12. Experimental Results
  13. Baseline Setup
  14. Performance Evaluation
  15. Ablation Analysis
  16. ...and 5 more sections

Key Result

theorem 1

Consider a simplified $L$-layer GNN $f_i$, where the update rule is $h_{v}^{l+1} = W^l_i \sum_{u \in \mathcal{N}_{v}} h_u^{l}$. The representation of a graph $G_{i,j}$ on client $i$ is given by $\sum_{v\in V_{i,j}} f(G_{i,j})_v = \Pi_i x^L_{i,j}$, where $x^L_{i,j} = \sum_{v\in \mathcal{V}_{i,j}} \ where $\bar{x}^L_i = \frac{1}{M_i} \sum_{j=1}^{M_i} x^L_{i, j}$.

Figures (5)

  • Figure 1: Graph data heterogeneity. \ref{['fig:fig1a']} Intra-domain heterogeneity involves two cases: first, clients possess data drawn from the same dataset but with inconsistent statistical properties; second, clients' data are sourced from different datasets within the same domain. \ref{['fig:fig1b']} Inter-domain heterogeneity implies that clients' data are derived from entirely different domains.
  • Figure 2: \ref{['fig:fig2a']} and \ref{['fig:fig2b']} show the representations generated by a randomly sampled client w/o and w/ representation decorrelation (RepDec) under the non-IID setting (COLLAB dataset), respectively. \ref{['fig:fig2c']} and \ref{['fig:fig2d']} show the representations under the inter-domain setting (BioSnCV), respectively. The representations are all generated by a three-layer graph attention network and then normalized into a unit ball.
  • Figure 3: Test accuracy ($\%$) at each communication round. Results are averaged over five runs. Shaded areas denote one standard deviation above and below the mean values.
  • Figure 4: The Loss landscapes of local GNN models for FedAvg and SEAL under different settings, where models are from the same client that is randomly selected.
  • Figure 5: Impact of regularizer coefficient $\alpha$.

Theorems & Definitions (1)

  • theorem 1