Federated Joint Learning for Domain and Class Generalization

TL;DR

This paper tackles the challenge of jointly generalizing to unseen classes and unseen domains in a federated setting for vision-language models. It introduces FedDCG, a framework with Domain-based Grouping, Class-Specific Domain-Grouping Training, and Domain-Guided Aggregation Inference, enabling class-generalized networks within domain groups and domain-aware inference via domain similarity. Key contributions include a novel domain-grouping strategy, a two-stage training procedure that decouples general and domain-specific knowledge, and a domain-guided aggregation mechanism that improves robustness across multiple datasets. Empirical results on Office-Home, MiniDomainNet, ImageNet-R, and ImageNet-A demonstrate state-of-the-art performance and resilience under low-data regimes, highlighting the practical impact for privacy-preserving, cross-domain, few-shot vision-language fine-tuning.

Abstract

Efficient fine-tuning of visual-language models like CLIP has become crucial due to their large-scale parameter size and extensive pretraining requirements. Existing methods typically address either the issue of unseen classes or unseen domains in isolation, without considering a joint framework for both. In this paper, we propose \textbf{Fed}erated Joint Learning for \textbf{D}omain and \textbf{C}lass \textbf{G}eneralization, termed \textbf{FedDCG}, a novel approach that addresses both class and domain generalization in federated learning settings. Our method introduces a domain grouping strategy where class-generalized networks are trained within each group to prevent decision boundary confusion. During inference, we aggregate class-generalized results based on domain similarity, effectively integrating knowledge from both class and domain generalization. Specifically, a learnable network is employed to enhance class generalization capabilities, and a decoupling mechanism separates general and domain-specific knowledge, improving generalization to unseen domains. Extensive experiments across various datasets show that \textbf{FedDCG} outperforms state-of-the-art baselines in terms of accuracy and robustness.

Figures (1)

• Figure 1: Illustration of our method. Our approach is a dual strategy: on the left, we group domains to learn class generalization networks, while on the right, we perform domain disentanglement pretraining.