Domain-Skewed Federated Learning with Feature Decoupling and Calibration

Abstract

Federated learning (FL) allows distributed clients to collaboratively train a global model in a privacy-preserving manner. However, one major challenge is domain skew, where clients' data originating from diverse domains may hinder the aggregated global model from learning a consistent representation space, resulting in poor generalizable ability in multiple domains. In this paper, we argue that the domain skew is reflected in the domain-specific biased features of each client, causing the local model's representations to collapse into a narrow low-dimensional subspace. We then propose Federated Feature Decoupling and Calibration ($F^2$DC), which liberates valuable class-relevant information by calibrating the domain-specific biased features, enabling more consistent representations across domains. A novel component, Domain Feature Decoupler (DFD), is first introduced in $F^2$DC to determine the robustness of each feature unit, thereby separating the local features into domain-robust features and domain-related features. A Domain Feature Corrector (DFC) is further proposed to calibrate these domain-related features by explicitly linking discriminative signals, capturing additional class-relevant clues that complement the domain-robust features. Finally, a domain-aware aggregation of the local models is performed to promote consensus among clients. Empirical results on three popular multi-domain datasets demonstrate the effectiveness of the proposed $F^2$DC and the contributions of its two modules. Code is available at https://github.com/mala-lab/F2DC.

Figures (8)

• Figure 1: Visualization of the singular values of the feature covariance matrix on the PACS li2017deeper dataset from Vanilla FL mcmahan2017communication (Top) and our method (Bottom). Under domain-skewed FL, the Vanilla method suffers from severe dimensional collapse, i.e., representations are largely biased and reside in a domain-specific lower-dimensional subspace. Our method effectively mitigates this collapse, as reflected in more uniformly distributed singular values and significantly fewer values tending to zero.
• Figure 2: Grad-CAM selvaraju2017grad visualization of the features obtained by three methods—Vanilla method mcmahan2017communication (Top), domain-skewed method FDSE wang2025federated (Middle), and $F^2$DC (Bottom)—for samples from different domains on a randomly selected $\text{Client}_{Photo}$ belonging to the photo domain. For our method $F^2$DC, we further provide a qualitative example about how its two modules work: local features are first decoupled via DFD into domain-robust features and domain-related features. Then, the domain-related features (which entangle valuable class-relevant information with domain biases) are calibrated via DFC, resulting in the Correction features. By incorporating the domain-robust and the corrected features, $F^2$DC captures more holistic class-relevant clues, enabling the local model to yield more consistent decisions for samples with the same semantics across different domains.
• Figure 3: Overview of the proposed $F^2$DC approach. During client-side local training, we initially separate local features into domain-robust features and domain-related features via a Domain Feature Decoupler (DFD) (Sec. \ref{['sec41']}), and then rectify the domain-related features via a Domain Feature Corrector (DFC) (Sec. \ref{['sec42']}). During server-side global aggregation, we perform domain-aware aggregation (Sec. \ref{['sec43']}) of the global model by incorporating each client's domain discrepancy $\mathbf{p}_{k}$ to promote more consensus among local clients.
• Figure 4: T-SNE visualization on PACS. Each color means one class, each shape means one domain, stars are semantic centers.
• Figure 5: Comparison of convergence in average accuracy on Office-Caltech (Left) and PACS (Right). Zoom in for details.