Hypernetwork-Driven Model Fusion for Federated Domain Generalization

TL;DR

The paper tackles Federated Domain Generalization (FDG) by addressing domain shifts in decentralized data via non-linear model aggregation. It introduces hFedF, a hypernetwork-based Federated Fusion framework that uses client-specific embeddings to generate personalized client parameters g_i = h(θ, ν[i]), enabling a quasi-global fusion without sharing raw data. Stability and convergence are achieved through GradAlign-based gradient alignment and Exponential Moving Average (EMA) regularization, with empirical validation on PACS, Office-Home, and VLCS showing superior in-domain and out-of-domain performance in zero-shot and few-shot settings. The results indicate strong robustness to domain heterogeneity, reduced weight divergence across clients, and improved prediction reliability, highlighting the practical potential of hypernetwork-based FDG for privacy-preserving, scalable domain generalization in FL.

Abstract

Federated Learning (FL) faces significant challenges with domain shifts in heterogeneous data, degrading performance. Traditional domain generalization aims to learn domain-invariant features, but the federated nature of model averaging often limits this due to its linear aggregation of local learning. To address this, we propose a robust framework, coined as hypernetwork-based Federated Fusion (hFedF), using hypernetworks for non-linear aggregation, facilitating generalization to unseen domains. Our method employs client-specific embeddings and gradient alignment techniques to manage domain generalization effectively. Evaluated in both zero-shot and few-shot settings, hFedF demonstrates superior performance in handling domain shifts. Comprehensive comparisons on PACS, Office-Home, and VLCS datasets show that hFedF consistently achieves the highest in-domain and out-of-domain accuracy with reliable predictions. Our study contributes significantly to the under-explored field of Federated Domain Generalization (FDG), setting a new benchmark for performance in this area.

Figures (13)

• Figure 1: (a) Examples of dog images from different domains in PACS dataset pacs. (b) An overview of FDG approaches: a) Domain-invariant feature extraction (domain alignment). b) Linear aggregation of models (aggregation alignment). c) Our method: non-linear hypernetwork fusion.
• Figure 2: a) Our hFedF framework with source domains ( \ref{['sec:method']} ). b) Domain splitting strategy: Visualization of splitting 3 domains across 2 clients, each holding samples from 2 domains ($d=2$). The largest domain is split into more parts to meet constraints. c) At inference, generalization is evaluated on an unseen target domain, and personalization is assessed on a held-out subset of the local source domain data.
• Figure 3: Comparison of id-accuracy (top row) and ood-accuracy (bottom row) across PACS pacs, Office-Home officehome, and VLCS vlcs datasets according to the changes of the number of shots.
• Figure 4: Performance comparison of id-accuracy (top row) and ood-accuracy (bottom row) on PACS dataset with different numbers of clients.
• Figure 5: (a) Convergence of gradient alignment weights, with aggregation weights depicted in transparent color per client over PACS ($d=1$). (b) Euclidean weight divergence fedprox between client models, showing the weight divergence across all clients, averaged per layer over PACS ($d=1$).