Tailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillation

TL;DR

Federated learning faces significant challenges from client heterogeneity and privacy constraints, especially under non-IID data. The authors propose a Tailored Federated Learning framework that combines model delta regularization (server-side delta estimation with an L2 penalty and historical delta tracking), personalized models with federated knowledge distillation, and mix-pooling readouts to handle diverse client tasks and preferences. The main contributions include a low-communication, delta-based update mechanism, KD-enabled personalization for extreme heterogeneity, and a flexible readout layer for graph data, all validated across FEMNIST, CIFAR-10, and CIKM22Cup showing improved convergence and accuracy. This approach offers a practical, bandwidth-efficient FL paradigm suitable for edge devices and privacy-sensitive domains such as healthcare, enabling robust performance despite diverse data and tasks.

Abstract

Federated learning (FL) has emerged as a transformative training paradigm, particularly invaluable in privacy-sensitive domains like healthcare. However, client heterogeneity in data, computing power, and tasks poses a significant challenge. To address such a challenge, we propose an FL optimization algorithm that integrates model delta regularization, personalized models, federated knowledge distillation, and mix-pooling. Model delta regularization optimizes model updates centrally on the server, efficiently updating clients with minimal communication costs. Personalized models and federated knowledge distillation strategies are employed to tackle task heterogeneity effectively. Additionally, mix-pooling is introduced to accommodate variations in the sensitivity of readout operations. Experimental results demonstrate the remarkable accuracy and rapid convergence achieved by model delta regularization. Additionally, the federated knowledge distillation algorithm notably improves FL performance, especially in scenarios with diverse data. Moreover, mix-pooling readout operations provide tangible benefits for clients, showing the effectiveness of our proposed methods.

Figures (5)

• Figure 1: Overview of federate knowledge distillation.
• Figure 2: Overview of the mix-pooling layer structure.
• Figure 3: FEMNIST Dataset Details
• Figure 4: Results of Fedr on FEMNIST and CIFAR-10.
• Figure 5: Results of Fedrkdp on the CIKM22Cup.