Robust Distributed Learning under Resource Constraints: Decentralized Quantile Estimation via (Asynchronous) ADMM

TL;DR

This work addresses robust decentralized learning on resource-constrained networks by introducing AsylADMM, an asynchronous gossip algorithm that estimates medians and quantiles with only two variables per node. The approach reformulates quantile estimation via pinball-loss-based M-estimation within an ADMM framework, achieving memory efficiency and fast empirical convergence. The authors provide convergence results for the synchronous variant and demonstrate substantial improvements over existing methods, along with applications to quantile/depth-based trimming and geometric-median estimation. Theoretical insights into rank-based trimming via Markov-chain concentration further bolster the method’s robustness, making AsylADMM well-suited for edge deployments in the presence of contaminated data.

Abstract

Specifications for decentralized learning on resource-constrained edge devices require algorithms that are communication-efficient, robust to data corruption, and lightweight in memory usage. While state-of-the-art gossip-based methods satisfy the first requirement, achieving robustness remains challenging. Asynchronous decentralized ADMM-based methods have been explored for estimating the median, a statistical centrality measure that is notoriously more robust than the mean. However, existing approaches require memory that scales with node degree, making them impractical when memory is limited. In this paper, we propose AsylADMM, a novel gossip algorithm for decentralized median and quantile estimation, primarily designed for asynchronous updates and requiring only two variables per node. We analyze a synchronous variant of AsylADMM to establish theoretical guarantees and empirically demonstrate fast convergence for the asynchronous algorithm. We then show that our algorithm enables quantile-based trimming, geometric median estimation, and depth-based trimming, with quantile-based trimming empirically outperforming existing rank-based methods. Finally, we provide a novel theoretical analysis of rank-based trimming via Markov chain theory.

Figures (13)

• Figure 1: Convergence performance of AsylADMM on contaminated Gaussian data. Plots (a) and (b) compare AsylADMM with existing optimization methods for median and quantile ($\alpha=0.3$) estimation. Plot (c) shows the impact graph topology on AsylADMM convergence, comparing Watts-Strogatz, geometric, and cycle graphs with varying connectivity levels. All plots show mean absolute error versus number of iterations, averaged over 100 trials with corresponding standard deviation.
• Figure 2: (a) Comparison of different optimization methods on geometric median estimation. (b) Comparison of quantile-based (via AsylADMM) versus rank-based (via GoRank) trimming for robust mean estimation. (c) Comparison of depth-based trimmed means (via GoDepth) with geometric median (via AsylADMM). All plots show mean absolute error versus number of iterations.
• Figure 3: Setup is the same as Plot (a) in Section 3 with different network sizes. In Plot (a), (b), and (c), we take $n=21$. In Plot (d), (e), and (f), we take $n=101$. In plot (g), (h), (i), the size is specified in the caption.
• Figure 4: Setup is the same as plot (b) in Section 3 with $n=101$ nodes on various graph topologies. Plots (a), (b), and (c) use a Geometric graph. Plots (d), (e), and (f) use a Watts-Strogatz. Plots (g), (h), and (i) use a Geometric graph.
• Figure 5: We use the same setup as in plot (c) of Section 3, varying only the data distribution.

Theorems & Definitions (16)

• Lemma 3.1
• proof
• Lemma 3.2
• proof
• Theorem 3.3
• proof
• Theorem 4.1: Exponential Bound
• Remark 4.2
• Lemma 5.1
• proof