FLAMMABLE: A Multi-Model Federated Learning Framework with Multi-Model Engagement and Adaptive Batch Sizes

TL;DR

FLAMMABLE addresses the inefficiencies of multi-model federated learning by jointly optimizing per-client batch sizes and enabling multi-model engagement to reduce round idle time. It introduces a principled batch-size adaptation method grounded in gradient-noise-based statistical progress and an ILP-based multi-model client selection to maximize global utility under deadlines. The framework is complemented by a novel MMFL benchmark platform and extensive experiments showing time-to-accuracy gains of up to 10× and final accuracy improvements up to 5.4% across diverse tasks. This work demonstrates that adaptive batching and cross-model participation can significantly accelerate MMFL convergence in heterogeneous environments, with practical implications for scalable, privacy-preserving distributed learning.

Abstract

Multi-Model Federated Learning (MMFL) is an emerging direction in Federated Learning (FL) where multiple models are trained in parallel, generally on various datasets. Optimizing the models' accuracies and training times in the MMFL setting requires adapting to data and system heterogeneity across clients as in single-model FL; these challenges are amplified in the MMFL setting due to additional heterogeneity across models. Neither existing solutions nor naïve extensions of single-model FL frameworks efficiently address these challenges. To bridge this gap, we propose FLAMMABLE, a comprehensive MMFL training framework. FLAMMABLE optimizes model training by intelligently adapting client batch sizes while engaging them to train multiple carefully chosen models, depending on their system capabilities, in each training round. To evaluate FLAMMABLE, we develop the first benchmark platform for the MMFL setting, which may enable future reproducible MMFL research. Extensive evaluations on multiple datasets and models show that FLAMMABLE boosts the MMFL time-to-accuracy performance by 1.1$\sim$10.0$\times$ while improving the final model accuracy by 1.3$\sim$5.4\% compared to several known baselines.

Figures (10)

• Figure 1: Overview of Multi-Model Federated Learning. Each client has the local data for training a subset of models. During every training round, the central server decides which client should train which model(s), shown by the darker-colored models in the figure. A faster client can train another model in the same round once it has finished its training tasks for other models.
• Figure 2: FLAMMABLE (1) reduces round duration by adapting the batch sizes (denoted by the height of each iteration): powerful devices can process an iteration with larger batch size with no/little increase in the time to process one iteration, (2) adapts the number of iterations per client to maintain training progress after batch size adaptation due to reduced statistical efficiency, and (3) allocates multiple models to fast clients to reduce idle time.
• Figure 3: Naïvely adapting client batch sizes without adapting the number of samples used in each model training round compromises training progress compared to using a constant batch size.
• Figure 4: Multi-model engagement, in which some clients may train multiple models in a single global training round, accelerates model training compared to single-model engagement.
• Figure 5: FLAMMABLE architecture and operation flow in each training round.