Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach

TL;DR

This work tackles the high communication cost of federated learning across multiple edge servers by introducing Confederated Learning (CFL) and a gradient-tracking-based method, CFL-SAGA, that employs a conditionally-triggered user selection (CTUS) mechanism. CTUS selectively uploads variance-reduced gradients from a small subset of users, balancing information gain with communication overhead, while maintaining a linear convergence rate. The authors establish convergence guarantees under standard smoothness and strong convexity assumptions, deriving a rate bound with a carefully chosen stepsize, and provide a theoretical analysis showing CTUS can prune non-informative uploads. Empirical results across different server topologies demonstrate substantial communication efficiency gains over state-of-the-art methods, validating the practical impact of the approach for scalable, privacy-preserving distributed learning.

Abstract

Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data dispersed over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as \emph{Confederated Learning} (CFL), in order to accommodate a larger number of users. A CFL system is composed of multiple networked edge servers, with each server connected to an individual set of users. Decentralized collaboration among servers is leveraged to harness all users' data for model training. Due to the potentially massive number of users involved, it is crucial to reduce the communication overhead of the CFL system. We propose a stochastic gradient method for distributed learning in the CFL framework. The proposed method incorporates a conditionally-triggered user selection (CTUS) mechanism as the central component to effectively reduce communication overhead. Relying on a delicately designed triggering condition, the CTUS mechanism allows each server to select only a small number of users to upload their gradients, without significantly jeopardizing the convergence performance of the algorithm. Our theoretical analysis reveals that the proposed algorithm enjoys a linear convergence rate. Simulation results show that it achieves substantial improvement over state-of-the-art algorithms in terms of communication efficiency.

Figures (7)

• Figure 1: The CFL framework with multiple servers.
• Figure 2: Topology of the server network
• Figure 3: Results on $\ell_2$-regularized logistic regression. First row: Optimality gap vs. number of iterations on different server networks; Second row: Optimality gap vs. communication overhead on different server networks
• Figure 4: Comparisons on $\ell_2$-regularized logistic regression. First row: Optimality gap vs. number of iterations on different server networks; Second row: Optimality gap vs. communication overhead on different server networks. SR is short for 'Sampling rate'.
• Figure 5: Runtime vs. number of iterations on the random graph

Theorems & Definitions (19)

• Definition 1
• Definition 2
• Theorem 1
• Proposition 1
• proof
• Lemma 1: Lemma 10 in QuLi17
• Lemma 2: Lemma 2 in PuShi20
• Lemma 3: Lemma 1 in XinKhan18
• Lemma 4: Corollary 8.1.29 in HornJohnson12
• Lemma 5: Lemma 4 in XinKhan20