Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations

TL;DR

This survey addresses the challenge of heterogeneity in federated learning by defining five axes of nonuniformity—data, model, task, device, and communication—and categorizing methods at data-, model-, and architecture-level, with a strong emphasis on privacy-preserving strategies. It compiles a broad set of techniques, including data cleaning and augmentation, parameter decoupling and compression, distillation, and multi-level architectures, to combat non-IID data, diverse models, and varying tasks across distributed clients. The work also details privacy threats (Byzantine, inference, etc.) and defenses (DP, SMC, and model tracing), highlighting the trade-offs between privacy, utility, and efficiency in heterogeneous FL deployments. By outlining future directions—toward fairness, secure collaboration, incentive mechanisms, and foundation-model integration—the paper provides a roadmap for building robust, private, and scalable FL systems in diverse real-world settings.

Abstract

In the field of heterogeneous federated learning (FL), the key challenge is to efficiently and collaboratively train models across multiple clients with different data distributions, model structures, task objectives, computational capabilities, and communication resources. This diversity leads to significant heterogeneity, which increases the complexity of model training. In this paper, we first outline the basic concepts of heterogeneous federated learning and summarize the research challenges in federated learning in terms of five aspects: data, model, task, device, and communication. In addition, we explore how existing state-of-the-art approaches cope with the heterogeneity of federated learning, and categorize and review these approaches at three different levels: data-level, model-level, and architecture-level. Subsequently, the paper extensively discusses privacy-preserving strategies in heterogeneous federated learning environments. Finally, the paper discusses current open issues and directions for future research, aiming to promote the further development of heterogeneous federated learning.

Figures (6)

• Figure 1: Illustration of five different skew patterns in data heterogeneity.
• Figure 2: FL task heterogeneity.
• Figure 3: FL model heterogeneity.
• Figure 4: FL communication heterogeneity.
• Figure 5: FL device heterogeneity.

Theorems & Definitions (1)

• Definition 1: $(\varepsilon, \delta)$-DP