Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Communication Efficient Collaborative Learning Framework for Distributed Features

Yang Liu, Yan Kang, Xinwei Zhang, Liping Li, Yong Cheng, Tianjian Chen, Mingyi Hong, Qiang Yang

TL;DR

This work tackles privacy-preserving collaborative learning when data are vertically partitioned across parties with different feature sets. It introduces Federated Stochastic Block Coordinate Descent (FedBCD), enabling multiple local updates before communication to greatly reduce rounds while preserving accuracy. The authors prove convergence guarantees under Lipschitz continuity and uniform sampling, showing an $\mathcal{O}(1/\sqrt{T})$ rate with $\mathcal{O}(\sqrt{T})$ communication rounds for appropriate choices of batch size, local iterations, and step size, with only mild dependence on the number of parties. Empirical results on MIMIC-III, MNIST, and NUS-WIDE demonstrate substantial communication savings and competitive performance, and a proximal variant further improves stability for large local updates; the approach is applicable to federated transfer learning with privacy-aware constraints.

Abstract

We introduce a collaborative learning framework allowing multiple parties having different sets of attributes about the same user to jointly build models without exposing their raw data or model parameters. In particular, we propose a Federated Stochastic Block Coordinate Descent (FedBCD) algorithm, in which each party conducts multiple local updates before each communication to effectively reduce the number of communication rounds among parties, a principal bottleneck for collaborative learning problems. We analyze theoretically the impact of the number of local updates and show that when the batch size, sample size, and the local iterations are selected appropriately, within $T$ iterations, the algorithm performs $\mathcal{O}(\sqrt{T})$ communication rounds and achieves some $\mathcal{O}(1/\sqrt{T})$ accuracy (measured by the average of the gradient norm squared). The approach is supported by our empirical evaluations on a variety of tasks and datasets, demonstrating advantages over stochastic gradient descent (SGD) approaches.

A Communication Efficient Collaborative Learning Framework for Distributed Features

TL;DR

This work tackles privacy-preserving collaborative learning when data are vertically partitioned across parties with different feature sets. It introduces Federated Stochastic Block Coordinate Descent (FedBCD), enabling multiple local updates before communication to greatly reduce rounds while preserving accuracy. The authors prove convergence guarantees under Lipschitz continuity and uniform sampling, showing an rate with communication rounds for appropriate choices of batch size, local iterations, and step size, with only mild dependence on the number of parties. Empirical results on MIMIC-III, MNIST, and NUS-WIDE demonstrate substantial communication savings and competitive performance, and a proximal variant further improves stability for large local updates; the approach is applicable to federated transfer learning with privacy-aware constraints.

Abstract

We introduce a collaborative learning framework allowing multiple parties having different sets of attributes about the same user to jointly build models without exposing their raw data or model parameters. In particular, we propose a Federated Stochastic Block Coordinate Descent (FedBCD) algorithm, in which each party conducts multiple local updates before each communication to effectively reduce the number of communication rounds among parties, a principal bottleneck for collaborative learning problems. We analyze theoretically the impact of the number of local updates and show that when the batch size, sample size, and the local iterations are selected appropriately, within iterations, the algorithm performs communication rounds and achieves some accuracy (measured by the average of the gradient norm squared). The approach is supported by our empirical evaluations on a variety of tasks and datasets, demonstrating advantages over stochastic gradient descent (SGD) approaches.

Paper Structure

This paper contains 15 sections, 4 theorems, 58 equations, 5 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Definition
  4. The FedSGD Approach
  5. The Proposed FedBCD Algorithms
  6. Security Analysis
  7. Security Definition
  8. Convergence Analysis
  9. Experiments
  10. Conclusions and Future Work
  11. Proof of Theorem 1
  12. Proof of Theorem 1
  13. Proof of Theorem 2
  14. Proof of Lemma \ref{['lemma_1']}
  15. Proof of Theorem 2

Key Result

Theorem 1

For $K$-party collaborative learning framework following eq:loss_linear with $K\geq 2$, the FedBCD Algorithm is secured for party $k$ if $k$'s feature dimension is larger than 2, i.e., $d_k \geq 2$.

Figures (5)

  • Figure 1: Illustration of a 2-party collaborative learning framework (a) with neural network(NN)-based local model. (b) FedBCD-s and FedBCD-p algorithms
  • Figure 2: Comparison of AUC (Left) and training loss (Right) in MIMIC-III dataset with varying local iterations, denoted by $Q$.
  • Figure 3: Comparison of AUC (Left) and training loss (Right) in NUS-WIDE dataset with varying local iterations, denoted by $L$.
  • Figure 4: The relationship between communication rounds and varying local iterations denoted by $Q$ for three target AUC (a), and the comparison between FedBCD-p and FedPBCD-p for large local iterations (b).
  • Figure 5: Comparison of AUC in MIMIC-III dataset with varying local iterations (denoted by $Q$) and number of parties (denoted by $K$). (Left) FedBCD-p; (Right) FedBCD-s

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Lemma 2