Fair Concurrent Training of Multiple Models in Federated Learning

TL;DR

The paper tackles fair concurrent training in multiple-model federated learning (MMFL), where heterogeneous task difficulty and uneven client participation can yield unfair training progress. It introduces FedFairMMFL, a difficulty-aware, $oldsymbol{ ext{α}}$-fair client-task allocation scheme, along with convergence guarantees, and couples this with incentive auctions to distribute client effort across tasks. The authors prove fairness advantages (variance reduction and improved uniformity) and provide convergence bounds that reflect allocation and participation dynamics, supported by empirical results. Experiments on multiple image and recognition tasks show improved minimum task accuracy and reduced performance variance while maintaining average performance, and the proposed auctions promote fair client recruitment under budget constraints. Overall, the work delivers a principled, theoretically grounded, and practically validated framework for fair, incentive-compatible MMFL in resource-constrained environments.

Abstract

Federated learning (FL) enables collaborative learning across multiple clients. In most FL work, all clients train a single learning task. However, the recent proliferation of FL applications may increasingly require multiple FL tasks to be trained simultaneously, sharing clients' computing and communication resources, which we call Multiple-Model Federated Learning (MMFL). Current MMFL algorithms use naive average-based client-task allocation schemes that can lead to unfair performance when FL tasks have heterogeneous difficulty levels, e.g., tasks with larger models may need more rounds and data to train. Just as naively allocating resources to generic computing jobs with heterogeneous resource needs can lead to unfair outcomes, naive allocation of clients to FL tasks can lead to unfairness, with some tasks having excessively long training times, or lower converged accuracies. Furthermore, in the FL setting, since clients are typically not paid for their training effort, we face a further challenge that some clients may not even be willing to train some tasks, e.g., due to high computational costs, which may exacerbate unfairness in training outcomes across tasks. We address both challenges by firstly designing FedFairMMFL, a difficulty-aware algorithm that dynamically allocates clients to tasks in each training round. We provide guarantees on airness and FedFairMMFL's convergence rate. We then propose a novel auction design that incentivizes clients to train multiple tasks, so as to fairly distribute clients' training efforts across the tasks. We show how our fairness-based learning and incentive mechanisms impact training convergence and finally evaluate our algorithm with multiple sets of learning tasks on real world datasets.

Figures (7)

• Figure 1: Fair Multiple Model Federated Learning to ensure that each model achieves comparable training performance: The pipeline's first part involves incentive-aware client recruitment auctions to ensure a fair take-up rate over tasks. This is followed by a difficulty-aware client-task allocation scheme that allocates tasks to clients based on the current loss levels of all tasks.
• Figure 2: Results of Experiment 1: 3 tasks with varying difficulty levels. The 3 tasks have datasets MNIST, CIFAR-10, and Fashion-MNIST. FedFairMMFL achieves a higher minimum accuracy across tasks, as compared to the baselines, while maintaining the same or higher accuracy for the other 2 tasks.
• Figure 3: Results of Experiment 2: Increasing the number of tasks. (a), (b), (c), and (d) compare the variance in accuracy observed across multiple tasks, when employing the different algorithms. The error bars denote the max/min values of the corresponding variance among all 4 random seeds. Our algorithm generally achieves a lower variance amongst task performance. (e), (f), and (g) show the trend of minimum accuracy when task numbers are 3, 6, and 10, respectively.
• Figure 4: Comparison of two alpha-fair based methods. a) 3 tasks include MNIST, CIFAR-10, and Fashion-MNIST. b) 4 tasks include MNIST, CIFAR-10, EMNIST, and Fashion-MNIST. EMNIST convergence is extremely slow with q-Fel. FedFairMMFL converges faster and achieves a higher minimum accuracy across tasks compared to q-Fel for task fairness.
• Figure 5: Results of Experiment 3: Increasing the number of clients. All experiments have 5 tasks (MNIST, CIFAR-10, Fashion-MNIST, EMNIST, and CIFAR-10). Experiments are conducted with 4 random seeds. The participation rate is 0.25, and client numbers are 80, 120, and 160, respectively. Our algorithm FedFairMMFL generally achieves a higher minimum accuracy, and/or a faster convergence.

Theorems & Definitions (17)

• Lemma 1: Fairness and Variance
• proof
• Lemma 2: Fairness and Cosine Similarity
• proof
• Lemma 3: Local and Global Model Discrepancy
• Theorem 4: Model Parameter Convergence
• Corollary 5: Fairness and Convergence
• proof
• Corollary 6: Model Convergence
• Lemma 7: Max-Min Fair Solution Algorithm