Modeling Inter-Intra Heterogeneity for Graph Federated Learning

TL;DR

This work tackles heterogeneity in Graph Federated Learning by introducing FedIIH, which jointly models inter- and intra-heterogeneity. Inter-heterogeneity is captured with HVGAE, a hierarchical variational graph autoencoder that infers subgraph data distributions and computes robust inter-subgraph similarities. Intra-heterogeneity is handled by disentangling each subgraph into $K$ latent factors with a DisenGCN-based approach, enabling per-factor federated aggregation. The optimization relies on a federated ELBO allowing parallel client updates, and the server aggregates parameters per latent factor using Jensen-Shannon divergence-based similarities. Experiments across 11 datasets show substantial gains, especially on heterophilic graphs, with FedIIH outperforming the second-best method by up to 5.79% and demonstrating strong robustness to varying hyperparameters.

Abstract

Heterogeneity is a fundamental and challenging issue in federated learning, especially for the graph data due to the complex relationships among the graph nodes. To deal with the heterogeneity, lots of existing methods perform the weighted federation based on their calculated similarities between pairwise clients (i.e., subgraphs). However, their inter-subgraph similarities estimated with the outputs of local models are less reliable, because the final outputs of local models may not comprehensively represent the real distribution of subgraph data. In addition, they ignore the critical intra-heterogeneity which usually exists within each subgraph itself. To address these issues, we propose a novel Federated learning method by integrally modeling the Inter-Intra Heterogeneity (FedIIH). For the inter-subgraph relationship, we propose a novel hierarchical variational model to infer the whole distribution of subgraph data in a multi-level form, so that we can accurately characterize the inter-subgraph similarities with the global perspective. For the intra-heterogeneity, we disentangle the subgraph into multiple latent factors and partition the model parameters into multiple parts, where each part corresponds to a single latent factor. Our FedIIH not only properly computes the distribution similarities between subgraphs, but also learns disentangled representations that are robust to irrelevant factors within subgraphs, so that it successfully considers the inter- and intra- heterogeneity simultaneously. Extensive experiments on six homophilic and five heterophilic graph datasets in both non-overlapping and overlapping settings demonstrate the effectiveness of our method when compared with nine state-of-the-art methods. Specifically, FedIIH averagely outperforms the second-best method by a large margin of 5.79% on all heterophilic datasets.

Figures (44)

• Figure 1: A framework comparison between existing methods and our FedIIH.
• Figure 2: The graphical model of our proposed HVGAE, where $\tilde{\mathbf{H}}_m^1, \tilde{\mathbf{H}}_m^2, \cdots, \tilde{\mathbf{H}}_m^K$ denote the local latent factors on the $m$-th client, and $\bm{\alpha}^1, \bm{\alpha}^2, \cdots, \bm{\alpha}^K$ denote the global latent factors on the server.
• Figure 3: Similarity heatmaps on the Amazon-ratings dataset in the overlapping setting with 20 clients.
• Figure 4: Since the intra-heterogeneity of the Tolokers dataset is high, the performances increase consistently as the value of $K$ increases.
• Figure 5: The architecture of our proposed HVGAE. The dashed lines show the process of sampling using the reparameterization trick kingma2013auto.