Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Iterative Sketching for Secure Coded Regression

Neophytos Charalambides, Hessam Mahdavifar, Mert Pilanci, Alfred O. Hero

TL;DR

The paper addresses speeding up distributed linear regression under straggler and security constraints by introducing iterative sketching with a random orthonormal projection and block sampling. This yields unbiased gradient estimates and a reduced-dimension problem that can be solved via distributed stochastic gradient descent without per-iteration encoding/decoding. Security is provided through information-theoretic guarantees when using a random projection from a finite subgroup of the orthogonal group, complemented by garbled block-SRHT for computational security. Empirical results demonstrate faster convergence and robustness to stragglers compared with traditional sketching methods, highlighting practical gains for secure, scalable distributed regression.

Abstract

Linear regression is a fundamental and primitive problem in supervised machine learning, with applications ranging from epidemiology to finance. In this work, we propose methods for speeding up distributed linear regression. We do so by leveraging randomized techniques, while also ensuring security and straggler resiliency in asynchronous distributed computing systems. Specifically, we randomly rotate the basis of the system of equations and then subsample blocks, to simultaneously secure the information and reduce the dimension of the regression problem. In our setup, the basis rotation corresponds to an encoded encryption in an approximate gradient coding scheme, and the subsampling corresponds to the responses of the non-straggling servers in the centralized coded computing framework. This results in a distributive iterative stochastic approach for matrix compression and steepest descent.

Iterative Sketching for Secure Coded Regression

TL;DR

The paper addresses speeding up distributed linear regression under straggler and security constraints by introducing iterative sketching with a random orthonormal projection and block sampling. This yields unbiased gradient estimates and a reduced-dimension problem that can be solved via distributed stochastic gradient descent without per-iteration encoding/decoding. Security is provided through information-theoretic guarantees when using a random projection from a finite subgroup of the orthogonal group, complemented by garbled block-SRHT for computational security. Empirical results demonstrate faster convergence and robustness to stragglers compared with traditional sketching methods, highlighting practical gains for secure, scalable distributed regression.

Abstract

Linear regression is a fundamental and primitive problem in supervised machine learning, with applications ranging from epidemiology to finance. In this work, we propose methods for speeding up distributed linear regression. We do so by leveraging randomized techniques, while also ensuring security and straggler resiliency in asynchronous distributed computing systems. Specifically, we randomly rotate the basis of the system of equations and then subsample blocks, to simultaneously secure the information and reduce the dimension of the regression problem. In our setup, the basis rotation corresponds to an encoded encryption in an approximate gradient coding scheme, and the subsampling corresponds to the responses of the non-straggling servers in the centralized coded computing framework. This results in a distributive iterative stochastic approach for matrix compression and steepest descent.
Paper Structure (33 sections, 24 theorems, 115 equations, 8 figures, 1 algorithm)

This paper contains 33 sections, 24 theorems, 115 equations, 8 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Coded Linear Regression
  3. Least Squares Approximation and Steepest Descent
  4. The Straggler Problem and Gradient Coding
  5. Secure Coded Computing Schemes
  6. The $\ell_2$-subspace embedding Property
  7. Properties of our Approach
  8. Block Subsampled Orthonormal Sketches
  9. Distributed Steepest Descent and Iterative Sketching
  10. Subspace Embedding of Algorithm \ref{['alg_orthog_sketch']}
  11. The Block-SRHT
  12. Subspace Embedding of the Block-SRHT
  13. Recursive Kronecker Products of Orthonormal Matrices
  14. Optimal Step-Size and Adaptive GC
  15. Security of Orthonormal Sketches
  16. ...and 18 more sections

Key Result

Lemma 1

At any iteration $t$, with no replications of the blocks across the network, the resulting sketching matrix $\bold{S}_{[t]}$ satisfies $\mathbb{E}\left[\bold{S}_{[t]}^T\bold{S}_{[t]}\right]=\mathbb{E}\left[\widetilde{\bold{\Omega}}_{[t]}^T\widetilde{\bold{\Omega}}_{[t]}\right]=\bold{I}_N$.

Figures (8)

  • Figure 1: Diagram of an exact coded matrix multiplication scheme, under the model proposed in LLPPR17. The central server sends an encoding $f_i(\bold{A},\bold{B})$ of matrices $\bold{A},\bold{B}$; usually of their submatrices, to each of the $n$ computational nodes. Then, each server performs a computation on their encoded data to obtain $\bold{W}_i$, which are then delivered to the central server. Once a certain fraction of servers have responded, the central server obtains $\bold{C}=\bold{A}\cdot\bold{B}$, after performing a decoding step. In this example, the second server is a straggler.
  • Figure 2: Illustration of our iterative sketching based GCS, at epoch $t+1$.
  • Figure 3: Flattening of block-scores, for $\bold{A}$ following a $t$-distribution. We abbreviate the garbled block-SRHT to 'G-b-SRHT'.
  • Figure 4: Illustration of how $\bold{P}$ and $\bold{D}$ modify the projection matrix $\hat{\bold{H}}_{64}$.
  • Figure 5: $\log$ residual error, for $\bold{A}$ following a $t$-distribution.
  • ...and 3 more figures

Theorems & Definitions (49)

  • Definition 1: Ch.2 KL14
  • Definition 2: Ch.3 KL14
  • Definition 3
  • Lemma 1
  • Theorem 1
  • Lemma 2
  • Remark 1
  • Corollary 1
  • Lemma 3
  • Theorem 2
  • ...and 39 more