FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonization

TL;DR

This work tackles domain shift and intra-/inter-class misalignment in federated semantic segmentation by introducing class exemplars to enforce class-consistency across global and local representations. The approach combines server-side global prototype supervision with client-side local adversarial harmonization, augmented by multilevel contrastive losses to unify two-level semantic spaces. Empirical results on multiple driving-scene datasets show FedSaaS outperforms state-of-the-art methods, with notable gains under severe heterogeneity and when generalizing to unseen domains. The method also demonstrates improved visualization of class embeddings and attention, and an analysis of communication and stability aspects. Overall, FedSaaS provides a principled, effective mechanism for robust, privacy-preserving semantic segmentation across heterogeneous clients.

Abstract

Federated semantic segmentation enables pixel-level classification in images through collaborative learning while maintaining data privacy. However, existing research commonly overlooks the fine-grained class relationships within the semantic space when addressing heterogeneous problems, particularly domain shift. This oversight results in ambiguities between class representation. To overcome this challenge, we propose a novel federated segmentation framework that strikes class consistency, termed FedSaaS. Specifically, we introduce class exemplars as a criterion for both local- and global-level class representations. On the server side, the uploaded class exemplars are leveraged to model class prototypes, which supervise global branch of clients, ensuring alignment with global-level representation. On the client side, we incorporate an adversarial mechanism to harmonize contributions of global and local branches, leading to consistent output. Moreover, multilevel contrastive losses are employed on both sides to enforce consistency between two-level representations in the same semantic space. Extensive experiments on several driving scene segmentation datasets demonstrate that our framework outperforms state-of-the-art methods, significantly improving average segmentation accuracy and effectively addressing the class-consistency representation problem.

Figures (13)

• Figure 1: (a) Data from different regions exhibits severe domain shift and label skew. In the semantic space, the same class often shows significantly different distributions across clients. (b) Existing FL methods primarily focus on generalization or personalization (left) but often ignore class-level alignment. Class-consistency FL (right) ensures consistent representation of the same class by aligning and constraining both global- and local-level representations.
• Figure 2: Overview of the FedSaaS Framework. (a) Client-side details. (b) Server-side and weighted integrated distillation details.
• Figure 3: Diagram of the prototype head and process of global prototype supervision.
• Figure 4: Comparison of visual results from different methods on datasets with severe heterogeneity.
• Figure 5: Visualization of the pixel embeddings for differenct semantic classes.