Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Random Alloy Codes and the Fundamental Limits of Coded Distributed Tensors

Pedro Soto

TL;DR

This paper focuses on the most likely event and measure the optimality of a coding scheme more directly by its probability of decoding, and leads to a practical construction of random codes for matrix multiplication, i.e., locally random alloy codes, which are optimal with respect to the measures.

Abstract

Tensors are a fundamental operation in distributed computing, \emph{e.g.,} machine learning, that are commonly distributed into multiple parallel tasks for large datasets. Stragglers and other failures can severely impact the overall completion time. Recent works in coded computing provide a novel strategy to mitigate stragglers with coded tasks, with an objective of minimizing the number of tasks needed to recover the overall result, known as the recovery threshold. However, we demonstrate that this strict combinatorial definition does not directly optimize the probability of failure. In this paper, we focus on the most likely event and measure the optimality of a coding scheme more directly by its probability of decoding. Our probabilistic approach leads us to a practical construction of random codes for matrix multiplication, i.e., locally random alloy codes, which are optimal with respect to the measures. Furthermore, the probabilistic approach allows us to discover a surprising impossibility theorem about both random and deterministic coded distributed tensors.

Random Alloy Codes and the Fundamental Limits of Coded Distributed Tensors

TL;DR

This paper focuses on the most likely event and measure the optimality of a coding scheme more directly by its probability of decoding, and leads to a practical construction of random codes for matrix multiplication, i.e., locally random alloy codes, which are optimal with respect to the measures.

Abstract

Tensors are a fundamental operation in distributed computing, \emph{e.g.,} machine learning, that are commonly distributed into multiple parallel tasks for large datasets. Stragglers and other failures can severely impact the overall completion time. Recent works in coded computing provide a novel strategy to mitigate stragglers with coded tasks, with an objective of minimizing the number of tasks needed to recover the overall result, known as the recovery threshold. However, we demonstrate that this strict combinatorial definition does not directly optimize the probability of failure. In this paper, we focus on the most likely event and measure the optimality of a coding scheme more directly by its probability of decoding. Our probabilistic approach leads us to a practical construction of random codes for matrix multiplication, i.e., locally random alloy codes, which are optimal with respect to the measures. Furthermore, the probabilistic approach allows us to discover a surprising impossibility theorem about both random and deterministic coded distributed tensors.
Paper Structure (8 sections, 6 theorems, 43 equations)

This paper contains 8 sections, 6 theorems, 43 equations.

Table of Contents

  1. Introduction
  2. Motivating examples: initial demonstration of alloy codes
  3. Background
  4. Code Constructions and Analysis
  5. Globally Random Alloy Codes for Order 2 Outer Products
  6. General Globally Random Alloy Codes of Order $\ell$
  7. An Impossibility Theorem
  8. Locally Random Alloy Codes

Key Result

Theorem 1

Suppose that $(g)^{1},...,(g)^{d}$ are i.i.d. uniformly random rank 1 tensors where $d = d_1\cdot ...\cdot d_\ell$, and $(g)^{k} \in \mathbb{F}^{d_1} \otimes ... \otimes \mathbb{F}^{d_\ell}$. Let $G$ be the matrix formed by flattening the $(g)^{i}$ and so that the $i^\mathrm{th}$ row of $G$ is $\mat if we use standard matrix notation. If $d_1+...+d_\ell \geq d_1\cdot ... \cdot d_\ell$, then we hav

Theorems & Definitions (12)

  • Theorem 1
  • proof
  • Theorem 2: Existence of Random Tensor Codes for Outer Products of Order 2
  • proof
  • Theorem 3: Existence of Random Tensor Codes for Outer Products of Order $\ell$
  • proof
  • Lemma 1
  • proof
  • Theorem 4
  • proof
  • ...and 2 more