Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal moments on redundancies in job cloning

Sahasrajit Sarmasarkar, Harish Pillai

TL;DR

This paper analyzes a master-server computation with $n$ jobs, $c$ child servers, where each job is replicated to $r$ servers and each server handles $k$ jobs. Under a uniform straggling model, it proves that the expected number of distinct completed jobs $d$ is the same across all balanced assignments and derives a variance expression that depends on how often job pairs share servers. It introduces the concept of a shape vector $h_D$ and shows proximally compact designs minimize variance while stretched compact designs maximize it, linking these extremal cases to generalized block designs (BIBDs) and replication schemes. The results extend to scenarios where the number of non-stragglers $x$ is random, using classic law of total variance, and have practical implications for redundancy strategies in straggler-prone distributed systems.

Abstract

We consider the problem of job assignment where a master server aims to compute some tasks and is provided a few child servers to compute under a uniform straggling pattern where each server is equally likely to straggle. We distribute tasks to the servers so that the master is able to receive most of the tasks even if a significant number of child servers fail to communicate. We first show that all \textit{balanced} assignment schemes have the same expectation on the number of distinct tasks received and then study the variance. We show constructions using a generalization of ``Balanced Incomplete Block Design''\cite{doi:10.1111/j.1469-1809.1939.tb02219.x,sprott1955} minimizes the variance, and constructions based on repetition coding schemes attain the largest variance.

Optimal moments on redundancies in job cloning

TL;DR

This paper analyzes a master-server computation with jobs, child servers, where each job is replicated to servers and each server handles jobs. Under a uniform straggling model, it proves that the expected number of distinct completed jobs is the same across all balanced assignments and derives a variance expression that depends on how often job pairs share servers. It introduces the concept of a shape vector and shows proximally compact designs minimize variance while stretched compact designs maximize it, linking these extremal cases to generalized block designs (BIBDs) and replication schemes. The results extend to scenarios where the number of non-stragglers is random, using classic law of total variance, and have practical implications for redundancy strategies in straggler-prone distributed systems.

Abstract

We consider the problem of job assignment where a master server aims to compute some tasks and is provided a few child servers to compute under a uniform straggling pattern where each server is equally likely to straggle. We distribute tasks to the servers so that the master is able to receive most of the tasks even if a significant number of child servers fail to communicate. We first show that all \textit{balanced} assignment schemes have the same expectation on the number of distinct tasks received and then study the variance. We show constructions using a generalization of ``Balanced Incomplete Block Design''\cite{doi:10.1111/j.1469-1809.1939.tb02219.x,sprott1955} minimizes the variance, and constructions based on repetition coding schemes attain the largest variance.
Paper Structure (20 sections, 17 theorems, 50 equations, 2 tables)

This paper contains 20 sections, 17 theorems, 50 equations, 2 tables.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Roadmap
  4. Related work
  5. Preliminaries and Notation
  6. The Mean and the Variance
  7. Results on extreme variance based on $[\mathfrak{m}^D(m)]_{m=0}^{r}$
  8. Minimal variance of $d$
  9. Maximal variance of $d$
  10. Number of non-stragglers ($x$) is random
  11. Conclusion
  12. Proof of Theorem \ref{['all_mean']}
  13. Computation of Variance of distinct jobs $\sigma_{D,x}(d)$
  14. Proof of Lemma \ref{['lemma_on_l_proximally_minimal']}
  15. Proof of Theorem \ref{['min_var']}
  16. ...and 5 more sections

Key Result

Theorem 1

Consider any balanced $(n,k,r,c)$ assignment $D$. The expectation of the number of distinct completed jobs $d$ received by the master when any subset of cardinality $x$ of the set of servers $\mathcal{S}$ is able to communicate with the master with equal probability is the same for every balanced $(

Theorems & Definitions (37)

  • Definition 1
  • Example 1
  • Theorem 1
  • proof : Proof Sketch
  • Lemma 1
  • Lemma 2
  • Definition 2
  • Definition 3
  • Definition 4
  • Lemma 3
  • ...and 27 more