Federated Communication-Efficient Multi-Objective Optimization

TL;DR

The paper tackles the challenge of federated multi-objective optimization (FMOO) where data are distributed across clients and multiple objective functions must be optimized jointly. It introduces FedCMOO, a communication-efficient FMOO algorithm that uses a single aggregated gradient per client, and a Gram-matrix-based weight computation that decouples communication from the number of objectives $M$. A convergence analysis shows that FedCMOO achieves Pareto-stationary convergence under mild smoothness and stochastic gradient assumptions, with improved dependence on $M$ compared to prior works; a variant, FedCMOO-Pref, enforces user-specified trade-offs via KL-divergence-based weighting. Empirically, FedCMOO and FedCMOO-Pref outperform baselines like FSMGDA and scalarized approaches across diverse datasets, while offering significantly reduced communication costs. These results demonstrate scalable, preference-aware FMOO with practical impact for federated settings such as healthcare and recommendation systems.

Abstract

We study a federated version of multi-objective optimization (MOO), where a single model is trained to optimize multiple objective functions. MOO has been extensively studied in the centralized setting but is less explored in federated or distributed settings. We propose FedCMOO, a novel communication-efficient federated multi-objective optimization (FMOO) algorithm that improves the error convergence performance of the model compared to existing approaches. Unlike prior works, the communication cost of FedCMOO does not scale with the number of objectives, as each client sends a single aggregated gradient to the central server. We provide a convergence analysis of the proposed method for smooth and non-convex objective functions under milder assumptions than in prior work. In addition, we introduce a variant of FedCMOO that allows users to specify a preference over the objectives in terms of a desired ratio of the final objective values. Through extensive experiments, we demonstrate the superiority of our proposed method over baseline approaches.

Figures (9)

• Figure 1: Mean test accuracy with $\text{MNIST+FMNIST}$ and $\text{CelebA}$ datasets. $\texttt{FedCMOO}$ outperforms $\texttt{FSMGDA}$ both in training speed and final accuracy.
• Figure 2: Mean test accuracy after 500 global rounds. As discussed earlier, with an increasing number of local iterations, the local objective drift of $\texttt{FSMGDA}$ leads to worse performance compared to $\texttt{FedCMOO}$.
• Figure 3: Mean test loss curves of $\texttt{FedCMOO}$, $\texttt{FedCMOO}\texttt{-Pref}$ (uniform preference), and $\texttt{FSMGDA}$ across 11 $\text{QM9}$ tasks. Left: Test loss vs. rounds. Right: Test loss vs. uploaded data (model size$\times$number of clients per round). $\texttt{FedCMOO}$ outperforms $\texttt{FSMGDA}$ due to reduced local drift, while both $\texttt{FedCMOO}$ and $\texttt{FedCMOO}\texttt{-Pref}$ achieve superior communication efficiency.
• Figure 4: $\texttt{FedCMOO}\texttt{-Pref}$ effectively finds solutions that align well with user-preferences in most cases. Imbalanced preferences or differences in the difficulty levels of objectives may cause misalignment.
• Figure 8: Mean test accuracy in $\text{MNIST+FMNIST}$, $\text{MultiMNIST}$, $\text{CIFAR10+FMNIST}$, $\text{CelebA}$, and $\text{CelebA-5}$ datasets. $\texttt{FedCMOO}$ outperforms the $\texttt{FSMGDA}$ in training speed and final accuracy. $\texttt{FedCMOO}\texttt{-Pref}$ with uniform preference either outperforms or is surpassed by $\texttt{FSMGDA}$ in terms of mean accuracy, but $\texttt{FedCMOO}\texttt{-Pref}$ trains the model for all objectives more uniformly across all tasks (see Table \ref{['tab:loss_L']} in the main text).

Theorems & Definitions (12)

• Proposition 1: Communication cost comparison
• Theorem 1: Convergence of $\texttt{FedCMOO}$
• Corollary 1.1: Convergence Rate
• Theorem 2: Restatement of \ref{['thm:FedCMO']}: Convergence of $\texttt{FedCMOO}$
• Corollary 2.1: Restatement of Corollary \ref{['cor:FedCMOO']}: Convergence Rate
• Lemma 1: Linear Algebra Tools
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5