Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

General Coded Computing in a Probabilistic Straggler Regime

Parsa Moradi, Mohammad Ali Maddah-Ali

TL;DR

This work addresses approximate general coded computing under a probabilistic straggler regime, where each server independently fails with probability $p$. It analyzes two general schemes, BACC and LeTCC, deriving upper bounds on the average approximation error and showing convergence to zero with rates at least $O\left(\log^3_{1/p}(N)/N^3\right)$ for LeTCC and $O\left(\log^4_{1/p}(N)/N^2\right)$ for BACC. The analysis combines a decomposition of error, Sobolev interpolation bounds, and probabilistic controls via the longest run of stragglers, and is validated on both one-dimensional and neural-network computing tasks. Experiments with $f_1(x)=x\sin(x)$ and a LeNet-5 model demonstrate faster convergence for LeTCC and support the claimed rates under realistic probabilistic straggler conditions.

Abstract

Coded computing has demonstrated promising results in addressing straggler resiliency in distributed computing systems. However, most coded computing schemes are designed for exact computation, requiring the number of responding servers to exceed a certain recovery threshold. Additionally, these schemes are tailored for highly structured functions. Recently, new coded computing schemes for general computing functions, where exact computation is replaced with approximate computation, have emerged. In these schemes, the availability of additional results corresponds to more accurate estimation of computational tasks. This flexibility introduces new questions that need to be addressed. This paper addresses the practically important scenario in the context of general coded computing, where each server may become a straggler with a probability $p$, independently from others. We theoretically analyze the approximation error of two existing general coded computing schemes: Berrut Approximate Coded Computing (BACC) and Learning Theoretic Coded Computing (LeTCC). Under the probabilistic straggler configuration, we demonstrate that the average approximation error for BACC and LeTCC converge to zero with the rate of at least $\mathcal{O}(\log^3_{\frac{1}{p}}(N)\cdot{N^{-3}})$ and $\mathcal{O}(\log^4_{\frac{1}{p}}(N)\cdot{N^{-2}})$, respectively. This is perhaps surprising, as earlier results does not indicate a convergence when the number of stragglers scales with the total number of servers $N$. However, in this case, despite the average number of stragglers being $Np$, the independence of servers in becoming stragglers allows the approximation error to converge to zero. These theoretical results are validated through experiments on various computing functions, including deep neural networks.

General Coded Computing in a Probabilistic Straggler Regime

TL;DR

This work addresses approximate general coded computing under a probabilistic straggler regime, where each server independently fails with probability . It analyzes two general schemes, BACC and LeTCC, deriving upper bounds on the average approximation error and showing convergence to zero with rates at least for LeTCC and for BACC. The analysis combines a decomposition of error, Sobolev interpolation bounds, and probabilistic controls via the longest run of stragglers, and is validated on both one-dimensional and neural-network computing tasks. Experiments with and a LeNet-5 model demonstrate faster convergence for LeTCC and support the claimed rates under realistic probabilistic straggler conditions.

Abstract

Coded computing has demonstrated promising results in addressing straggler resiliency in distributed computing systems. However, most coded computing schemes are designed for exact computation, requiring the number of responding servers to exceed a certain recovery threshold. Additionally, these schemes are tailored for highly structured functions. Recently, new coded computing schemes for general computing functions, where exact computation is replaced with approximate computation, have emerged. In these schemes, the availability of additional results corresponds to more accurate estimation of computational tasks. This flexibility introduces new questions that need to be addressed. This paper addresses the practically important scenario in the context of general coded computing, where each server may become a straggler with a probability , independently from others. We theoretically analyze the approximation error of two existing general coded computing schemes: Berrut Approximate Coded Computing (BACC) and Learning Theoretic Coded Computing (LeTCC). Under the probabilistic straggler configuration, we demonstrate that the average approximation error for BACC and LeTCC converge to zero with the rate of at least and , respectively. This is perhaps surprising, as earlier results does not indicate a convergence when the number of stragglers scales with the total number of servers . However, in this case, despite the average number of stragglers being , the independence of servers in becoming stragglers allows the approximation error to converge to zero. These theoretical results are validated through experiments on various computing functions, including deep neural networks.

Paper Structure

This paper contains 12 sections, 6 theorems, 44 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Coded Computing for Straggler Resistance
  4. Coded Computing Frameworks for General Computing Function
  5. Problem Formulation
  6. Main Results
  7. Proof Sketch
  8. Experimental Results
  9. Proof of Theorem \ref{['th:letcc']}
  10. Proof of Theorem \ref{['th:berrut']}
  11. Proof of Corollary \ref{['cor:rate']}
  12. Proof of Corollary \ref{['cor:cheb']}

Key Result

Theorem 1

Consider the $\texttt{LeTCC}$ framework consisting of $N$ servers, where $u_\textrm{enc}(\cdot), u_\textrm{dec}(\cdot) \in \mathcal{H}^{2}\left(\Omega\right)$ and $\lambda_\textrm{dec} \leqslant N^{-4}$. Let $f(\cdot)$ be a $\nu$-Lipschitz function satisfying $|f"| \leq \eta$ for some $\eta > 0$. As

Figures (1)

  • Figure 1: Log-log plot showing the error convergence rates of $\texttt{LeTCC}$ and $\texttt{BACC}$ for the functions $f(x) = x\sin(x)$ and the LeNet5 network, with $p = 0.05$ and $p = 0.1$ (top). The plot also includes a comparison between $p = 0.1$ and the non-probabilistic configuration with $S = 2$ (bottom).

Theorems & Definitions (7)

  • Theorem 1
  • Theorem 2
  • Corollary 1
  • Remark 1
  • Corollary 2
  • Lemma 1
  • Lemma 2