Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generation

TL;DR

HypeMeFed tackles federated learning with heterogeneous clients by merging a depth-wise multi-exit network with server-side hypernetworks that generate missing deeper-layer weights. The approach uses a low-rank factorization via SVD to compress hypernetwork weight generation, enabling scalable parameter generation and reducing memory and computation on servers. Empirical results across multiple datasets and a real-world embedded testbed show improved accuracy over FedAvg variants, substantial reductions in hypernetwork overhead, and practical latency improvements, validating the method for inclusive participation of diverse devices. The work contributes a concrete design, implementation details, and open-source code to enable deployment of heterogeneous FL with efficient hypernetwork-based weight generation.

Abstract

While federated learning leverages distributed client resources, it faces challenges due to heterogeneous client capabilities. This necessitates allocating models suited to clients' resources and careful parameter aggregation to accommodate this heterogeneity. We propose HypeMeFed, a novel federated learning framework for supporting client heterogeneity by combining a multi-exit network architecture with hypernetwork-based model weight generation. This approach aligns the feature spaces of heterogeneous model layers and resolves per-layer information disparity during weight aggregation. To practically realize HypeMeFed, we also propose a low-rank factorization approach to minimize computation and memory overhead associated with hypernetworks. Our evaluations on a real-world heterogeneous device testbed indicate that \system enhances accuracy by 5.12% over FedAvg, reduces the hypernetwork memory requirements by 98.22%, and accelerates its operations by 1.86x compared to a naive hypernetwork approach. These results demonstrate HypeMeFed's effectiveness in leveraging and engaging heterogeneous clients for federated learning.

Figures (16)

• Figure 1: Parameter aggregation in heterogeneous federated learning showing information disparity.
• Figure 2: Hypernetwork-based weight generation.
• Figure 3: Accuracy, CKA, and weight value distribution plots for different federated learning configurations and potential solutions proposed in this work.
• Figure 4: Visualized samples of trained, generated, and randomly initialized parameters.
• Figure 5: Overview of HypeMeFed. HypeMeFed leverages heterogeneous models generated from a multi-exit network architecture with hypernetworks to resolve the per-layer information disparity issue. Best viewed in color.