Seamless Integration: Sampling Strategies in Federated Learning Systems

TL;DR

This work addresses the challenge of seamlessly integrating new clients into federated learning systems under data non-IIDness and heterogeneous resources. It investigates clustering-based client selection, validating approaches on a proprietary manufacturing dataset and an ImageNet subset by analyzing final-layer activations to form client clusters. The study finds that agglomerative clustering, particularly with complete linkage and a higher cluster count, delivers the best early- and mid-round accuracy and stability compared with random or loss-based selection, with manageable computational overhead in cross-silo settings. The findings support deploying clustering-based client onboarding to improve robustness, efficiency, and scalability of production FL in manufacturing and similar domains.

Abstract

Federated Learning (FL) represents a paradigm shift in the field of machine learning, offering an approach for a decentralized training of models across a multitude of devices while maintaining the privacy of local data. However, the dynamic nature of FL systems, characterized by the ongoing incorporation of new clients with potentially diverse data distributions and computational capabilities, poses a significant challenge to the stability and efficiency of these distributed learning networks. The seamless integration of new clients is imperative to sustain and enhance the performance and robustness of FL systems. This paper looks into the complexities of integrating new clients into existing FL systems and explores how data heterogeneity and varying data distribution (not independent and identically distributed) among them can affect model training, system efficiency, scalability and stability. Despite these challenges, the integration of new clients into FL systems presents opportunities to enhance data diversity, improve learning performance, and leverage distributed computational power. In contrast to other fields of application such as the distributed optimization of word predictions on Gboard (where federated learning once originated), there are usually only a few clients in the production environment, which is why information from each new client becomes all the more valuable. This paper outlines strategies for effective client selection strategies and solutions for ensuring system scalability and stability. Using the example of images from optical quality inspection, it offers insights into practical approaches. In conclusion, this paper proposes that addressing the challenges presented by new client integration is crucial to the advancement and efficiency of distributed learning networks, thus paving the way for the adoption of Federated Learning in production environments.

Figures (6)

• Figure 1: Illustration of the federated learning process Legler.2023.
• Figure 2: Miniature Truck, captured by an inline quality inspection module. Only a cabin with one type of windshield (red) is pictured. In the following step, one trailer is added from a selection of options.
• Figure 3: Comparison of clustering methods on test accuracy across 100 communication rounds.
• Figure 4: Comparison of client participation frequencies across different selection methods: The chart displays the number of participations for each client using three selection strategies: Random selection, highest training loss and cluster selection with k=3.
• Figure 5: The plot shows the moving average of test accuracy across 200 communication rounds for three client selection methods: Random selection, training loss-based selection, and agglomerative clustering, each with their corresponding parameters.