Decentralized Sparse Linear Regression via Gradient-Tracking: Linear Convergence and Statistical Guarantees

TL;DR

This work develops a decentralized sparse linear regression method over mesh networks using gradient-tracking (DGT). It proves linear convergence to within centralized statistical precision $O(s\log d/N)$ under high-dimensional scaling, leveraging global restricted strong convexity and smoothness (RSC/RSM) and local RSM conditions, with rates depending on network connectivity $\rho$. A novel high-dimensional tracking-error analysis and a $z$-transform framework enable contraction guarantees despite non-strong convexity and dimension growth $d/N\to\infty$, with explicit iteration and communication complexities. Under random designs, the results hold with high probability, showing that distributed computation can match centralized statistical performance when the network is sufficiently connected. The work also demonstrates practical scalability and favorable communication-efficiency tradeoffs across graph topologies, validated by extensive numerics and real-data experiments.

Abstract

We study sparse linear regression over a network of agents, modeled as an undirected graph and no server node. The estimation of the $s$-sparse parameter is formulated as a constrained LASSO problem wherein each agent owns a subset of the $N$ total observations. We analyze the convergence rate and statistical guarantees of a distributed projected gradient tracking-based algorithm under high-dimensional scaling, allowing the ambient dimension $d$ to grow with (and possibly exceed) the sample size $N$. Our theory shows that, under standard notions of restricted strong convexity and smoothness of the loss functions, suitable conditions on the network connectivity and algorithm tuning, the distributed algorithm converges globally at a {\it linear} rate to an estimate that is within the centralized {\it statistical precision} of the model, $O(s\log d/N)$. When $s\log d/N=o(1)$, a condition necessary for statistical consistency, an $\varepsilon$-optimal solution is attained after $\mathcal{O}(κ\log (1/\varepsilon))$ gradient computations and $O (κ/(1-ρ) \log (1/\varepsilon))$ communication rounds, where $κ$ is the restricted condition number of the loss function and $ρ$ measures the network connectivity. The computation cost matches that of the centralized projected gradient algorithm despite having data distributed; whereas the communication rounds reduce as the network connectivity improves. Overall, our study reveals interesting connections between statistical efficiency, network connectivity \& topology, and convergence rate in high dimensions.

Figures (8)

• Figure 1: Two network architectures: master-worker and mesh networks.
• Figure 2: Some commonly used graphs ${\mathcal{G}}$ for the communication network. Dashed line in the Erdős-Rényi graph stands for the existence of an edge with some probability.
• Figure 3: Scalability of DGT for fixed networks. The network is composed of $m = 50$ agents with connectivity $\rho = 0.0638$. Dashed lines indicate estimation error if using $\widehat{{\boldsymbol{\theta}}}$.
• Figure 4: Plot of the log-normalized estimation error for $m \in \{50, 625, 1250, 2500\}.$ In this problem $d = 5000,$$s = \lceil \sqrt{d} \rceil,$$n \in \{50,4,2,1\}$ and $N = 2500.$ All curves in each panel are obtained using the same data points. Black horizontal lines indicate the normalized estimation error if using $\widehat{{\boldsymbol{\theta}}}.$
• Figure 5: Number of communication rounds required for each network to satisfy $\rho^k \leq m^{-8}$.

Theorems & Definitions (28)

• Remark 5
• Theorem 6
• Corollary 7
• Theorem 9
• Lemma 10: agarwal2012fast
• Proposition 11
• proof
• Proposition 12
• proof
• Proposition 13