Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion-based Decentralized Federated Multi-Task Representation Learning

Donghwa Kang, Shana Moothedath

TL;DR

This work addresses decentralized multi-task representation learning (MTRL) in high-dimensional, data-scarce regimes by learning a shared low-rank representation $oldsymbol{U}^igstar$ together with task-specific coefficients $oldsymbol{b}_t$. It introduces Dif-AltGDmin, a diffusion-based alternating gradient descent and minimization algorithm that exchanges only subspace estimates across a connected network, thereby reducing communication compared to centralized or prior decentralized methods. The authors provide constructive, high-probability guarantees on subspace recovery and sample/iteration complexity, and demonstrate improved time and communication efficiency over prior work, including robustness to sparse network connectivity. Empirical results on synthetic data corroborate that Dif-AltGDmin attains near-centralized performance with substantially lower communication overhead. The approach advances practical decentralized learning for MTRL and offers a foundation for privacy-preserving, scalable representation learning in distributed systems.

Abstract

Representation learning is a widely adopted framework for learning in data-scarce environments to obtain a feature extractor or representation from various different yet related tasks. Despite extensive research on representation learning, decentralized approaches remain relatively underexplored. This work develops a decentralized projected gradient descent-based algorithm for multi-task representation learning. We focus on the problem of multi-task linear regression in which multiple linear regression models share a common, low-dimensional linear representation. We present an alternating projected gradient descent and minimization algorithm for recovering a low-rank feature matrix in a diffusion-based decentralized and federated fashion. We obtain constructive, provable guarantees that provide a lower bound on the required sample complexity and an upper bound on the iteration complexity of our proposed algorithm. We analyze the time and communication complexity of our algorithm and show that it is fast and communication-efficient. We performed numerical simulations to validate the performance of our algorithm and compared it with benchmark algorithms.

Diffusion-based Decentralized Federated Multi-Task Representation Learning

TL;DR

This work addresses decentralized multi-task representation learning (MTRL) in high-dimensional, data-scarce regimes by learning a shared low-rank representation together with task-specific coefficients . It introduces Dif-AltGDmin, a diffusion-based alternating gradient descent and minimization algorithm that exchanges only subspace estimates across a connected network, thereby reducing communication compared to centralized or prior decentralized methods. The authors provide constructive, high-probability guarantees on subspace recovery and sample/iteration complexity, and demonstrate improved time and communication efficiency over prior work, including robustness to sparse network connectivity. Empirical results on synthetic data corroborate that Dif-AltGDmin attains near-centralized performance with substantially lower communication overhead. The approach advances practical decentralized learning for MTRL and offers a foundation for privacy-preserving, scalable representation learning in distributed systems.

Abstract

Representation learning is a widely adopted framework for learning in data-scarce environments to obtain a feature extractor or representation from various different yet related tasks. Despite extensive research on representation learning, decentralized approaches remain relatively underexplored. This work develops a decentralized projected gradient descent-based algorithm for multi-task representation learning. We focus on the problem of multi-task linear regression in which multiple linear regression models share a common, low-dimensional linear representation. We present an alternating projected gradient descent and minimization algorithm for recovering a low-rank feature matrix in a diffusion-based decentralized and federated fashion. We obtain constructive, provable guarantees that provide a lower bound on the required sample complexity and an upper bound on the iteration complexity of our proposed algorithm. We analyze the time and communication complexity of our algorithm and show that it is fast and communication-efficient. We performed numerical simulations to validate the performance of our algorithm and compared it with benchmark algorithms.
Paper Structure (6 sections, 7 theorems, 39 equations, 2 figures, 3 algorithms)

This paper contains 6 sections, 7 theorems, 39 equations, 2 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Notations and Problem Formulation
  3. Diffusion-based Decentralized MTRL
  4. Proof of Theorem \ref{['thm: main']}
  5. Simulations
  6. Conclusion

Key Result

Proposition 1

(olshevsky2009convergence) Consider the agreement algorithm in Algorithm alg:AvgCons with doubly stochastic weight matrix $\boldsymbol{W}$. Let $z_\mathrm{true}:=\frac{1}{L}\sum_{g=1}^L z^{\hbox{(in)}}_{g}$ be the true average of the initial values $z^{\hbox{(in)}}_{g}$ across $L$ nodes. For any $\e Proposition prop: avgcons is stated for scalar consensus, but it naturally extends to matrix valued

Figures (2)

  • Figure 1: Subspace distance vs. iteration count and execution time in seconds. We compare the performance of algorithms by varying number of consensus iterations, $T_{\mathrm{con}}$. In all cases $T_{\mathrm{GD}}=500$, $L=20$, $d=T=600,\ r=4,\ n=30$, and $p=0.5$.
  • Figure 2: Subspace distance vs. iteration count and execution time. We compare the performance of our algorithm by varying the edge probability in the communication graph, $p$. In all cases $T_{\mathrm{con,GD}}=T_{\mathrm{con,init}}=10, T_{\mathrm{GD}}=1500$, $L=d=T=100, r=10$, and $n=50$.

Theorems & Definitions (8)

  • Proposition 1
  • Proposition 2
  • proof
  • Theorem 1
  • Proposition 3
  • Lemma 1
  • Lemma 2
  • Lemma 3