Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Byzantine-Resilient Federated PCA and Low Rank Column-wise Sensing

Ankit Pratap Singh, Namrata Vaswani

TL;DR

The paper tackles Byzantine-resilient federated learning for two linked problems: PCA and horizontally federated LRCS. It introduces Subspace-Median, a geometric-median-based, communication- and sample-efficient subspace estimator, and shows it enables attack-robust federated PCA with per-node samples $q_\ell \ge C n r$ and favorable communication costs. For LRCS, it develops Byz-AltGDmin, a complete algorithm with GM-based initialization and gradient aggregation that achieves $\epsilon$-accurate recovery with per-node samples $\mathcal{O}(n r^2 \log(1/\epsilon))$ and communication $\mathcal{O}(n r \log(1/\epsilon))$, extending the GM framework to nonconvex federated LR problems. The methods leverage a subspace distance metric, robust GM lemmas, and Davis–Kahan-type analyses to provide non-asymptotic guarantees and show strong empirical performance under Byzantine attacks. The work advances practical, Byzantine-resilient federated learning for low-rank recovery tasks and offers extendable techniques for other LR problems and subspace estimation tasks in distributed settings.

Abstract

This work considers two related learning problems in a federated attack prone setting: federated principal components analysis (PCA) and federated low rank column-wise sensing (LRCS). The node attacks are assumed to be Byzantine which means that the attackers are omniscient and can collude. We introduce a novel provably Byzantine-resilient communication-efficient and sampleefficient algorithm, called Subspace-Median, that solves the PCA problem and is a key part of the solution for the LRCS problem. We also study the most natural Byzantine-resilient solution for federated PCA, a geometric median based modification of the federated power method, and explain why it is not useful. Our second main contribution is a complete alternating gradient descent (GD) and minimization (altGDmin) algorithm for Byzantine-resilient horizontally federated LRCS and sample and communication complexity guarantees for it. Extensive simulation experiments are used to corroborate our theoretical guarantees. The ideas that we develop for LRCS are easily extendable to other LR recovery problems as well.

Byzantine-Resilient Federated PCA and Low Rank Column-wise Sensing

TL;DR

The paper tackles Byzantine-resilient federated learning for two linked problems: PCA and horizontally federated LRCS. It introduces Subspace-Median, a geometric-median-based, communication- and sample-efficient subspace estimator, and shows it enables attack-robust federated PCA with per-node samples and favorable communication costs. For LRCS, it develops Byz-AltGDmin, a complete algorithm with GM-based initialization and gradient aggregation that achieves -accurate recovery with per-node samples and communication , extending the GM framework to nonconvex federated LR problems. The methods leverage a subspace distance metric, robust GM lemmas, and Davis–Kahan-type analyses to provide non-asymptotic guarantees and show strong empirical performance under Byzantine attacks. The work advances practical, Byzantine-resilient federated learning for low-rank recovery tasks and offers extendable techniques for other LR problems and subspace estimation tasks in distributed settings.

Abstract

This work considers two related learning problems in a federated attack prone setting: federated principal components analysis (PCA) and federated low rank column-wise sensing (LRCS). The node attacks are assumed to be Byzantine which means that the attackers are omniscient and can collude. We introduce a novel provably Byzantine-resilient communication-efficient and sampleefficient algorithm, called Subspace-Median, that solves the PCA problem and is a key part of the solution for the LRCS problem. We also study the most natural Byzantine-resilient solution for federated PCA, a geometric median based modification of the federated power method, and explain why it is not useful. Our second main contribution is a complete alternating gradient descent (GD) and minimization (altGDmin) algorithm for Byzantine-resilient horizontally federated LRCS and sample and communication complexity guarantees for it. Extensive simulation experiments are used to corroborate our theoretical guarantees. The ideas that we develop for LRCS are easily extendable to other LR recovery problems as well.
Paper Structure (78 sections, 21 theorems, 103 equations, 1 figure, 4 tables, 7 algorithms)

This paper contains 78 sections, 21 theorems, 103 equations, 1 figure, 4 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Existing Work
  3. Byzantine-resilient federated machine learning (ML)
  4. Work on robust PCA and subspace learning and tracking, and other robust estimation problems
  5. Work on robust statistics - robust mean and robust covariance estimation
  6. Work on the LR Column-wise Sensing (LRCS) problem
  7. Federated PCA and subspace learning; no attacks
  8. Our Contributions
  9. Novelty of our algorithmic and proof techniques
  10. Organization
  11. Problem set-up, Notation, and Geometric Median Preliminaries
  12. Problem setup
  13. Resilient Federated Subspace Estimation
  14. Resilient Federated PCA
  15. Resilient Horizontally-Federated Low Rank Column-wise Sensing (LRCS)
  16. ...and 63 more sections

Key Result

Lemma 3.1

For a $\delta > 0$, consider Algorithm NEW_PCA_1 with $T_{\tiny{GM}} = C\log \left(\frac{Lr}{\delta}\right)$. Assume that Assumption byzassu holds. Assume that, for at least $(1-\tau)L$ nodes, the following holds: Then, w.p. at least $1- c_{\tiny{\text{approxGM}}} - \exp(-L \psi(0.4-\tau,p) )$, Here $\psi(a, b):=(1-a)\log \frac{1-a}{1-b}+a\log \frac{a}{b}$ for $0 < a,b < 1$ is the binary KL dive

Figures (1)

  • Figure 1: Byz-AltGDmin (Median) vs Byz-AltGDmin (MoM) for $L_{byz}=1,2$; $L=18$

Theorems & Definitions (40)

  • Definition 1
  • Claim 2.1: Theorem 1 cohen2016geometric
  • Lemma 3.1: Subspace-Median
  • proof
  • Remark 3.2
  • Theorem 3.3: Subspace-Median guarantee
  • proof
  • Theorem 3.4: ResPowMeth guarantee
  • proof
  • Corollary 4.1: Subspace Median for PCA
  • ...and 30 more