Jointly Computation- and Communication-Efficient Distributed Learning

TL;DR

This work addresses distributed learning over networks by designing LT-ADMM-CC, a novel algorithm that blends computation- and communication-efficiency. It integrates stochastic gradients with variance reduction, multi-epoch local training, compression with error feedback, and a local-update scheme to reduce communication rounds, and it proves exact linear convergence in the strongly convex setting. The approach is validated via numerical experiments on a classification task, where LT-ADMM-CC outperforms state-of-the-art compressed-communication methods by delivering exact convergence with lower time costs. The results underscore the practical impact of jointly optimizing computation and communication in distributed learning systems.

Abstract

We address distributed learning problems over undirected networks. Specifically, we focus on designing a novel ADMM-based algorithm that is jointly computation- and communication-efficient. Our design guarantees computational efficiency by allowing agents to use stochastic gradients during local training. Moreover, communication efficiency is achieved as follows: i) the agents perform multiple training epochs between communication rounds, and ii) compressed transmissions are used. We prove exact linear convergence of the algorithm in the strongly convex setting. We corroborate our theoretical results by numerical comparisons with state of the art techniques on a classification task.

Figures (2)

• Figure 1: LT-ADMM-CC with different compressors.
• Figure 2: Comparison of distributed optimization algorithms with compressed communication.

Theorems & Definitions (9)

• Theorem 1
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof
• Lemma 4
• proof