KnFu: Effective Knowledge Fusion

TL;DR

The KnFu is a personalized effective knowledge fusion scheme for each client, that analyzes effectiveness of different local models’ knowledge prior to the aggregation phase, that outperforms its FL-based, FKD-based, and local training baselines in scenarios where the clients have small datasets with intermediate degree of heterogeneity.

Abstract

Federated Learning (FL) has emerged as a prominent alternative to the traditional centralized learning approach. Generally speaking, FL is a decentralized approach that allows for collaborative training of Machine Learning (ML) models across multiple local nodes, ensuring data privacy and security while leveraging diverse datasets. Conventional FL, however, is susceptible to gradient inversion attacks, restrictively enforces a uniform architecture on local models, and suffers from model heterogeneity (model drift) due to non-IID local datasets. To mitigate some of these challenges, the new paradigm of Federated Knowledge Distillation (FKD) has emerged. FDK is developed based on the concept of Knowledge Distillation (KD), which involves extraction and transfer of a large and well-trained teacher model's knowledge to lightweight student models. FKD, however, still faces the model drift issue. Intuitively speaking, not all knowledge is universally beneficial due to the inherent diversity of data among local nodes. This calls for innovative mechanisms to evaluate the relevance and effectiveness of each client's knowledge for others, to prevent propagation of adverse knowledge. In this context, the paper proposes Effective Knowledge Fusion (KnFu) algorithm that evaluates knowledge of local models to only fuse semantic neighbors' effective knowledge for each client. The KnFu is a personalized effective knowledge fusion scheme for each client, that analyzes effectiveness of different local models' knowledge prior to the aggregation phase. Comprehensive experiments were performed on MNIST and CIFAR10 datasets illustrating effectiveness of the proposed KnFu in comparison to its state-of-the-art counterparts. A key conclusion of the work is that in scenarios with large and highly heterogeneous local datasets, local training could be preferable to knowledge fusion-based solutions.

Figures (4)

• Figure 1: The EPD as an estimation of the class distribution.
• Figure 2: Illustration of data heterogeneity among $10$ clients on the CIFAR-10 dataset.
• Figure 3: Learning curves of the ALMA metric corresponding to different methods, for a fixed heterogeneity level, $\alpha= 0.5$, and different local data sizes, $|\mathbb D_n| = \{50, 100, 200\}$.
• Figure 4: Learning curves of the ALMA metric associated with different methods, for a fixed local data size, $|\mathbb D_n| = 100$, and various heterogeneity levels, $\alpha= \{0.1, 0.25, 0.5, 1$}.