FedCova: Robust Federated Covariance Learning Against Noisy Labels

TL;DR

FedCova is proposed, a dependency-free federated covariance learning framework that eliminates external reliances by enhancing the model's intrinsic robustness via a new perspective on feature covariances, and unifies three key processes through the covariance.

Abstract

Noisy labels in distributed datasets induce severe local overfitting and consequently compromise the global model in federated learning (FL). Most existing solutions rely on selecting clean devices or aligning with public clean datasets, rather than endowing the model itself with robustness. In this paper, we propose FedCova, a dependency-free federated covariance learning framework that eliminates such external reliances by enhancing the model's intrinsic robustness via a new perspective on feature covariances. Specifically, FedCova encodes data into a discriminative but resilient feature space to tolerate label noise. Built on mutual information maximization, we design a novel objective for federated lossy feature encoding that relies solely on class feature covariances with an error tolerance term. Leveraging feature subspaces characterized by covariances, we construct a subspace-augmented federated classifier. FedCova unifies three key processes through the covariance: (1) training the network for feature encoding, (2) constructing a classifier directly from the learned features, and (3) correcting noisy labels based on feature subspaces. We implement FedCova across both symmetric and asymmetric noisy settings under heterogeneous data distribution. Experimental results on CIFAR-10/100 and real-world noisy dataset Clothing1M demonstrate the superior robustness of FedCova compared with the state-of-the-art methods.

Figures (13)

• Figure 1: Overview of the FedCova framework. The green data is clean, while the red data is mislabeled. The pink rectangle indicates covariance-aware feature learning. The gray rectangle with the circle of blue arrows - indicates the conventional FL processes, around which the circle of red arrows - in FedCova constructs a fortress to guard against label noise under the flow of the covariances $\boldsymbol{\Sigma}$. Specifically, the server first broadcasts the global model and global classifier to edge devices. Label correction can be conducted then, after which devices perform local feature learning and update the local models and local classifiers to the server. The latter then aggregates them for the next rounds of iterations.
• Figure 2: Effect of adding error tolerance term to the covariance matrix on the probability distributions.
• Figure 3: Effect of $\epsilon^2$ illustrated as sample distribution and axis projections.
• Figure 4: Non-i.i.d. data distribution among devices.
• Figure 5: Visualization of different bilevel noise settings.

Theorems & Definitions (1)

• Remark 3.1