Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning

TL;DR

The proposed framework offers useful guidance on the suitability of various data divergence measures in FL systems and is beneficial for keeping related research activities on the right track in terms of designing PFL schemes and selecting appropriate data heterogeneity evaluation approaches.

Abstract

There is growing research interest in measuring the statistical heterogeneity of clients' local datasets. Such measurements are used to estimate the suitability for collaborative training of personalized federated learning (PFL) models. Currently, these research endeavors are taking place in silos and there is a lack of a unified benchmark to provide a fair and convenient comparison among various approaches in common settings. We aim to bridge this important gap in this paper. The proposed benchmarking framework currently includes six representative approaches. Extensive experiments have been conducted to compare these approaches under five standard non-IID FL settings, providing much needed insights into which approaches are advantageous under which settings. The proposed framework offers useful guidance on the suitability of various data divergence measures in FL systems. It is beneficial for keeping related research activities on the right track in terms of: (1) designing PFL schemes, (2) selecting appropriate data heterogeneity evaluation approaches for specific FL application scenarios, and (3) addressing fairness issues in collaborative model training. The code is available at https://github.com/Xiaoni-61/DH-Benchmark.

Figures (12)

• Figure 1: The decision tree for identifying the optimal FL scheme under different non-IID setting.
• Figure 2: The change of test accuracy with the number of clients: the first (resp. last) two plots are for CIFAR-10 (resp. MNIST).
• Figure 3: The training curves of different approaches on MNIST.
• Figure 4: The training curves of different approaches on FMNIST.
• Figure 5: The training curves of different approaches on SVHN.