(Corrected Version) Push-LSVRG-UP: Distributed Stochastic Optimization over Unbalanced Directed Networks with Uncoordinated Triggered Probabilities

TL;DR

This work introduces Push-LSVRG-UP, a variance-reduced distributed stochastic optimization method designed for unbalanced directed networks. By marrying the push-sum consensus mechanism with loopless SVRG and an uncoordinated probabilistic triggering scheme, the algorithm achieves linear convergence to the global optimum with an explicit step-size interval and favorable iteration complexity, all while reducing per-iteration storage costs. Theoretical guarantees are complemented by two real-data experiments (distributed logistic regression and MNIST-based SVM), which demonstrate accelerated convergence and practical scalability compared with existing approaches. The approach broadens the applicability of stochastic distributed optimization to general directed networks, preserving privacy and lowering computational burden per agent. These results offer a principled framework for efficient large-scale distributed learning over heterogeneous communication topologies with flexible, asynchronous gradient computations.

Abstract

Distributed stochastic optimization, arising in the crossing and integration of traditional stochastic optimization, distributed computing and storage, and network science, has advantages of high efficiency and a low per-iteration computational complexity in resolving large-scale optimization problems. This paper concentrates on resolving a large-scale convex finite-sum optimization problem in a multi-agent system over unbalanced directed networks. To tackle this problem in an efficient way, a distributed consensus optimization algorithm, adopting the push-sum technique and a distributed loopless stochastic variance-reduced gradient (LSVRG) method with uncoordinated triggered probabilities, is developed and named Push-LSVRG-UP. Each agent under this algorithmic framework performs only local computation and communicates only with its neighbors without leaking their private information. The convergence analysis of Push-LSVRG-UP is relied on analyzing the contraction relationships between four error terms associated with the multi-agent system. Theoretical results provide an explicit feasible range of the constant step-size, a linear convergence rate, and an iteration complexity of Push-LSVRG-UP when achieving the globally optimal solution. It is shown that Push-LSVRG-UP achieves the superior characteristics of accelerated linear convergence, fewer storage costs, and a lower per-iteration computational complexity than most existing works. Meanwhile, the introduction of an uncoordinated probabilistic triggered mechanism allows Push-LSVRG-UP to facilitate the independence and flexibility of agents in computing local batch gradients. In simulations, the practicability and improved performance of Push-LSVRG-UP are manifested via resolving two distributed learning problems based on real-world datasets.

Figures (14)

• Figure 1: Directed network (a) vs undirected network (b).
• Figure 2: An unbalanced directed network with $m=30$.
• Figure 3: Performance comparison over epochs.
• Figure 4: Performance comparison over CPU running time.
• Figure 5: Exponential unbalanced directed networks with different agents.

Theorems & Definitions (25)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Theorem 1
• proof
• Remark 5
• Remark 6
• Lemma 1
• proof