Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift

TL;DR

FedCCFA, a federated learning framework with classifier clustering and feature alignment that adaptively aligns clients' feature spaces based on the entropy of label distribution $P(Y)$, alleviating the inconsistency in feature space is proposed.

Abstract

Data heterogeneity is one of the key challenges in federated learning, and many efforts have been devoted to tackling this problem. However, distributed concept drift with data heterogeneity, where clients may additionally experience different concept drifts, is a largely unexplored area. In this work, we focus on real drift, where the conditional distribution $P(Y|X)$ changes. We first study how distributed concept drift affects the model training and find that local classifier plays a critical role in drift adaptation. Moreover, to address data heterogeneity, we study the feature alignment under distributed concept drift, and find two factors that are crucial for feature alignment: the conditional distribution $P(Y|X)$ and the degree of data heterogeneity. Motivated by the above findings, we propose FedCCFA, a federated learning framework with classifier clustering and feature alignment. To enhance collaboration under distributed concept drift, FedCCFA clusters local classifiers at class-level and generates clustered feature anchors according to the clustering results. Assisted by these anchors, FedCCFA adaptively aligns clients' feature spaces based on the entropy of label distribution $P(Y)$, alleviating the inconsistency in feature space. Our results demonstrate that FedCCFA significantly outperforms existing methods under various concept drift settings. Code is available at https://github.com/Chen-Junbao/FedCCFA.

Figures (9)

• Figure 1: The impact of distributed concept drift on model training. Distributed concept drift occurs at round 100. Decoupled: classifier-then-extractor learning method. Decoupled-Clustering: Decoupled method with classifier clustering.
• Figure 2: An overview of the proposed FedCCFA. Clients train balanced classifiers and local models, and then generate local anchors. The server performs client clustering with the help of balanced classifiers, and then aggregates local models and local anchors.
• Figure 3: Data partition visualization under 20 clients with full participation.
• Figure 4: Data partition visualization under 100 clients with 20% participation.
• Figure 5: Ablation study on scaling factor $\gamma$. $\gamma = 20$ is an optimal selection.