FedKDX: Federated Learning with Negative Knowledge Distillation for Enhanced Healthcare AI Systems

TL;DR

FedKDX advances federated healthcare AI by introducing Negative Knowledge Distillation (NKD) alongside contrastive learning and adaptive gradient compression to address non-IID data and privacy constraints. The approach unifies traditional knowledge distillation with semantic and structural alignment, yielding improved accuracy, AUC, and convergence across SLEEP, UCI-HAR, and PAMAP2, with robustness under limited client participation. Empirical results show up to 2.53% accuracy gains and AUC scores above 0.98, while dynamic SVD-based gradient compression reduces communication overhead. The authors provide a thorough ablation and analysis of NKD, CTL, and SVD, discuss SVD non-convergence challenges, and propose future work to extend validation to broader healthcare modalities and larger-scale deployments.

Abstract

This paper introduces FedKDX, a federated learning framework that addresses limitations in healthcare AI through Negative Knowledge Distillation (NKD). Unlike existing approaches that focus solely on positive knowledge transfer, FedKDX captures both target and non-target information to improve model generalization in healthcare applications. The framework integrates multiple knowledge transfer techniques--including traditional knowledge distillation, contrastive learning, and NKD--within a unified architecture that maintains privacy while reducing communication costs. Through experiments on healthcare datasets (SLEEP, UCI-HAR, and PAMAP2), FedKDX demonstrates improved accuracy (up to 2.53% over state-of-the-art methods), faster convergence, and better performance on non-IID data distributions. Theoretical analysis supports NKD's contribution to addressing statistical heterogeneity in distributed healthcare data. The approach shows promise for privacy-sensitive medical applications under regulatory frameworks like HIPAA and GDPR, offering a balanced solution between performance and practical implementation requirements in decentralized healthcare settings. The code and model are available at https://github.com/phamdinhdat-ai/Fed_2024.

Figures (7)

• Figure 1: Overview of the FedKDX workflow. This cross-device collaborative learning framework integrates knowledge distillation and gradient factorization to enable efficient federated learning for HAR. The process iterates through three phases: (1) local training with distillation from the global model, (2) gradient compression via dynamic SVD, and (3) global model refinement through aggregated updates. This cycle balances model personalization with collective learning while minimizing communication overhead.
• Figure 2: Schematic diagram of the CNN-based model architecture. The model features two CONV blocks (with batch normalization and max pooling) and two FC layers designed for human activity recognition (HAR).
• Figure 3: Data distribution of the SLEEP dataset (16 participants, 12 postures). (Left) Total training and testing samples per client. (Right) Distribution of postures across participants, showing the balanced data allocation for sleep monitoring.
• Figure 4: Data distribution of the UCI-HAR dataset (30 subjects, 6 daily activities). (Left) Training and testing sample sizes per user. (Right) Class distribution across individuals, demonstrating the balanced nature of the controlled mobile sensing environment.
• Figure 5: Data distribution of the PAMAP2 dataset (9 subjects, 18 physical activities). (Left) Total samples per subject for training and testing. (Right) Activity coverage across users, reflecting the varying data density typical of multi-sensor activity recognition.