Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach

TL;DR

This work tackles the challenges of federated graph foundation models by addressing semantic-structural misalignment, cross-domain locality, and bandwidth constraints. It proposes FedGALA, a two-phase framework that first federatedly pre-trains a decoupled graph encoder and frozen PLM using unsupervised contrastive learning and history-aware updates, then employs lightweight prompt tuning for task adaptation. The key innovations include continuous graph–text alignment, history-matching for stability, class-wise semantic tokens, and group-aware prompt aggregation to preserve domain specialization. Empirical results across multiple TAG datasets and tasks show that FedGALA outperforms a broad range of baselines, including other FedGFMs, with notable gains in few-shot settings and improved efficiency. The approach offers a practical, bandwidth-efficient path to robust, cross-domain graph understanding in privacy-preserving, distributed environments.

Abstract

Recent studies of federated graph foundational models (FedGFMs) break the idealized and untenable assumption of having centralized data storage to train graph foundation models, and accommodate the reality of distributed, privacy-restricted data silos. Despite their simplicity and intuition, existing studies that project aligned generalizable knowledge onto a discrete token space via vector-quantized backbones suffer from irreversible knowledge loss during the quantization process. In this context, we argue that reconciling the semantic-structural orthogonality and integrity between pre-trained language models (PLMs) and graph neural networks (GNNs) is paramount for developing effective FedGFMs while simultaneously mitigating the severe data heterogeneity and communication constraints inherent in distributed, resource-limited environments. To address these issues, we propose FedGALA (Federated Graph And Language Alignment), a framework that resolves graph-based semantic-structural orthogonality and integrity in federated settings by employing unsupervised contrastive learning to align GNNs and frozen PLMs within a continuous embedding space, thereby capturing robust, transferable general knowledge. Subsequently, FedGALA leverages a communication-efficient prompt tuning mechanism to steer these pre-aligned encoders and frozen PLMs, facilitating effective adaptation to diverse downstream tasks while circumventing the prohibitive overhead of full-parameter fine-tuning. The comprehensive experiments validate that FedGALA outperforms all competitive baselines across multi-domain datasets on multiple tasks with up to 14.37% performance improvement.

Figures (3)

• Figure 1: FedGALA framework contains two training phases: ❶ Federated Pre-training Graph Encoder: aligning frozen PLMs with partitioned graph encoders (structural and semantic) via contrastive alignment and history-matching to derive global structural parameters, class-wise tokens, and local semantic encoders. ❷ Local Prompt-based Federated Fine-tuning: adapting the frozen PLMs to various downstream tasks, and the trained soft prompts and graph prompts are consolidated via group-aware prompt aggregation.
• Figure 2: Sensitivity analysis of FedGALA.
• Figure 3: Convergence rates during the pre-training phase, where the proposed FedGALA consistently achieves faster convergence than all baseline methods.