Beyond Centralization: Provable Communication Efficient Decentralized Multi-Task Learning

TL;DR

This work tackles decentralized multi-task representation learning under data scarcity, aiming to recover a rank-$r$ shared representation ${\boldsymbol{U}}$ and task-specific coefficients ${\boldsymbol{b}_t}$ without a fusion center. It introduces Dif-AltGDmin, a diffusion-based, alternating projected gradient descent and least-squares minimization algorithm that operates across a connected network, exchanging only subspace estimates ${\boldsymbol{U}}$, not raw data. The authors provide rigorous guarantees: with high probability, they achieve $\text{SD}_2( {\boldsymbol{U}}^{(T_{\text{GD}})}, {\boldsymbol{U}^*}) \le \epsilon$ and $\|\boldsymbol{\theta}_t^{(T_{\text{GD}})} - \boldsymbol{\theta}_t^*\| \le 1.4\epsilon\|\boldsymbol{\theta}_t^*\|$, given suitable initialization and sample counts, and they characterize time, communication, and sample complexities. A key highlight is that the communication complexity is independent of the target accuracy $\epsilon$, leading to substantial reductions compared with prior centralized or diffusion-based methods. Simulations demonstrate the method’s robustness to sparse networks and show regimes where decentralized learning can outperform centralized federated approaches in data-scarce settings.

Abstract

Representation learning is a widely adopted framework for learning in data-scarce environments, aiming to extract common features from related tasks. While centralized approaches have been extensively studied, decentralized methods remain largely underexplored. We study decentralized multi-task representation learning in which the features share a low-rank structure. We consider multiple tasks, each with a finite number of data samples, where the observations follow a linear model with task-specific parameters. In the decentralized setting, task data are distributed across multiple nodes, and information exchange between nodes is constrained by a communication network. The goal is to recover the underlying feature matrix whose rank is much smaller than both the parameter dimension and the number of tasks. We propose a new alternating projected gradient and minimization algorithm with provable accuracy guarantees. We provide comprehensive characterizations of the time, communication, and sample complexities. Importantly, the communication complexity is independent of the target accuracy, which significantly reduces communication cost compared to prior methods. Numerical simulations validate the theoretical analysis across different dimensions and network topologies, and demonstrate regimes in which decentralized learning outperforms centralized federated approaches.

Figures (4)

• Figure 1: Subspace distance vs. iteration count and execution time in seconds. In all plots, y-axis is $\mathrm{SD}_2(\boldsymbol{U}_{1}^{\hbox{$(\tau)$}},{\boldsymbol{U}^\star})$. We compare the performance of algorithms by under different communication settings -- $T_{\mathrm{con}}:=T_{\mathrm{con,init}}=T_{\mathrm{con,GD}}$, $p$, and $L$. The default setting is $T_{\mathrm{con}}=5$, $T_{\mathrm{GD}}=200$, $L=300$, $d=300,\ T=800,\ r=4,\ n=50$, and $p=0.03$, and we vary one parameter at a time while fixing all other parameters at their default values.
• Figure 2: Subspace distance vs. iteration count and execution time. In all plots, y-axis is $\mathrm{SD}_2(\boldsymbol{U}_{1}^{\hbox{$(\tau)$}},{\boldsymbol{U}^\star})$. We compare the impact of problem parameter sizes-- $d,\ r,$ and $T$. The default setting is $T_{\mathrm{con,GD}}=T_{\mathrm{con,init}}=5, T_{\mathrm{GD}}=400$, $L=20$, $d=T=600, r=4$, and $n=50$, and we vary one parameter at a time while fixing all other parameters at their default values.
• Figure :
• Figure :

Theorems & Definitions (15)

• Proposition 1
• Proposition 2
• proof
• Theorem 1
• Remark 1
• Proposition 3
• Proposition 4
• Proposition 5
• Proposition 6
• Lemma 1