Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning

TL;DR

This work addresses semantic degradation in Federated Prototype Learning under data and model heterogeneity by introducing FedTSP, which injects textual semantics into prototypes. A server-side LLM generates fine-grained class descriptions, which are encoded into textual prototypes by a PLM with trainable prompts to bridge the image-text gap. A contrastive alignment loss guides local clients to learn from these semantic prototypes, yielding faster convergence and improved accuracy across GFL, PFL, and HtFL settings. Extensive experiments on CIFAR-10/100 and Tiny ImageNet show consistent gains over state-of-the-art prototype-based methods, with robustness to heterogeneity and reduced communication overhead due to compact prototype exchange.

Abstract

Federated Prototype Learning (FedPL) has emerged as an effective strategy for handling data heterogeneity in Federated Learning (FL). In FedPL, clients collaboratively construct a set of global feature centers (prototypes), and let local features align with these prototypes to mitigate the effects of data heterogeneity. The performance of FedPL highly depends on the quality of prototypes. Existing methods assume that larger inter-class distances among prototypes yield better performance, and thus design different methods to increase these distances. However, we observe that while these methods increase prototype distances to enhance class discrimination, they inevitably disrupt essential semantic relationships among classes, which are crucial for model generalization. This raises an important question: how to construct prototypes that inherently preserve semantic relationships among classes? Directly learning these relationships from limited and heterogeneous client data can be problematic in FL. Recently, the success of pre-trained language models (PLMs) demonstrates their ability to capture semantic relationships from vast textual corpora. Motivated by this, we propose FedTSP, a novel method that leverages PLMs to construct semantically enriched prototypes from the textual modality, enabling more effective collaboration in heterogeneous data settings. We first use a large language model (LLM) to generate fine-grained textual descriptions for each class, which are then processed by a PLM on the server to form textual prototypes. To address the modality gap between client image models and the PLM, we introduce trainable prompts, allowing prototypes to adapt better to client tasks. Extensive experiments demonstrate that FedTSP mitigates data heterogeneity while significantly accelerating convergence.

Figures (8)

• Figure 1: Visualization of the cosine similarity between prototypes of different classes on the CIFAR-10 for various methods. (a) AlignFed uniformly distributes prototypes on a hypersphere, leading to equal similarity between all class pairs and the absence of meaningful semantic relationships. (b) FedTGP maximizes the inter-class distance during prototype training, resulting in low similarity between classes and failing to capture meaningful inter-class relationships. (c) FedProto directly aggregates client prototypes to build a global prototype, enabling it to learn semantic relationships from client data. However, due to limited and heterogeneous client data, FedProto learns incorrect relationships, such as trucks and deer being more similar than trucks and automobiles. (d) Our method constructs prototypes using LLMs and PLMs from the textual modality, effectively learning more accurate semantic relationships, such as higher similarity between animals than between animals and vehicles.
• Figure 2: Overview of FedTSP.
• Figure 3: Comparison of local and global accuracy on CIFAR-10 and CIFAR-100 under the Dir(0.5) settings using $\text{HtFE}_9$.
• Figure 4: Performance comparison of Top-1 and Top-5 accuracy on CIFAR-100 under Dir(0.5) using $\text{HtFE}_3$ for different methods.
• Figure 5: Left: Effect of prompt type for text encoder. Right: Effect of local epochs $E_c$ on different methods.