FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneity

TL;DR

This paper proposes a dual-clustered feature contrast-based FL framework with dual focuses that facilitates cross-client knowledge sharing by pulling the local representation closer to clusters shared by clients with similar semantics while pushing them away from clusters with dissimilar semantics.

Abstract

Federated learning (FL) facilitates a privacy-preserving neural network training paradigm through collaboration between edge clients and a central server. One significant challenge is that the distributed data is not independently and identically distributed (non-IID), typically including both intra-domain and inter-domain heterogeneity. However, recent research is limited to simply using averaged signals as a form of regularization and only focusing on one aspect of these non-IID challenges. Given these limitations, this paper clarifies these two non-IID challenges and attempts to introduce cluster representation to address them from both local and global perspectives. Specifically, we propose a dual-clustered feature contrast-based FL framework with dual focuses. First, we employ clustering on the local representations of each client, aiming to capture intra-class information based on these local clusters at a high level of granularity. Then, we facilitate cross-client knowledge sharing by pulling the local representation closer to clusters shared by clients with similar semantics while pushing them away from clusters with dissimilar semantics. Second, since the sizes of local clusters belonging to the same class may differ for each client, we further utilize clustering on the global side and conduct averaging to create a consistent global signal for guiding each local training in a contrastive manner. Experimental results on multiple datasets demonstrate that our proposal achieves comparable or superior performance gain under intra-domain and inter-domain heterogeneity.

Figures (6)

• Figure 1: Illustration of four challenges in FL. Figure (a) illustrates balanced intra-domain heterogeneity, where samples are drawn from the same domain but exhibit label shifts with equal quantities. Figure (b) shows imbalanced intra-domain heterogeneity, where samples are drawn from the same domain but with varying quantity and label shifts. Figure (c) denotes balanced inter-domain heterogeneity, where samples are drawn from different domains but with the same sample quantities. Figure (d) represents imbalanced inter-domain heterogeneity, where samples are drawn from different domains with varying sample quantities.
• Figure 2: Illustration of our proposed FedCCL framework. The proposed framework comprises five main steps: local training, local clustering, local signal aggregation, global signal aggregation and averaging, and global model parameter averaging. In this paper, we focus on steps 2 to 4.
• Figure 3: Comparison of communication efficiency (%) on CIFAR-10 task under imbalanced intra-domain heterogeneity scenarios.
• Figure 4: Comparison of communication efficiency (%) on DomainNet task under balanced inter-domain heterogeneity scenarios.
• Figure 5: Comparison of average test accuracy (%) on DomainNet under imbalanced inter-domain heterogeneity scenarios. A lower Dirichlet parameter value indicates server heterogeneity.