Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal and Asymptotically Optimal Data and Task Allocation in Distributed Computing

Javad Maheri, K. K. Krishnan Namboodiri, Petros Elia

TL;DR

The paper addresses the joint optimization of communication and computation in a master–worker distributed setting where a function decomposes into $d$-wise subfunctions over a library of $n$ files. It introduces the deterministic Interweaved-Cliques (IC) design, which partitions the $d$-uniform hypergraph of dependencies into $N$ groups to achieve $\pi_{\mathbf{X}} \asymp n / N^{1/d}$ and keep $\delta_{\mathbf{X}}$ bounded (typically $\le 5$) with high probability, for a broad class of decompositions $\mathbf{X}$. The main theoretical contributions include a tight lower bound $\pi_{\mathbf{X}}^{*} \ge \varphi^{1/d} n / N^{1/d}$ and an IC-based achievability showing order-optimal partitioning gain $\propto N^{1/d}$, plus corollaries that extend optimality across multiple decompositions and degrees. Practically, the IC design provides a universal, reshuffle-free data placement that remains valid across many function representations, enabling efficient multi-function computation in distributed systems with parallel communication channels.

Abstract

We study the joint minimization of communication and computation costs in distributed computing, where a master node coordinates $N$ workers to evaluate a function over a library of $n$ files. Assuming that the function is decomposed into an arbitrary subfunction set $\mathbf{X}$, with each subfunction depending on $d$ input files, renders our distributed computing problem into a $d$-uniform hypergraph edge partitioning problem wherein the edge set (subfunction set), defined by $d$-wise dependencies between vertices (files) must be partitioned across $N$ disjoint groups (workers). The aim is to design a file and subfunction allocation, corresponding to a partition of $\mathbf{X}$, that minimizes the communication cost $π_{\mathbf{X}}$, representing the maximum number of distinct files per server, while also minimizing the computation cost $δ_{\mathbf{X}}$ corresponding to a maximal worker subfunction load. For a broad range of parameters, we propose a deterministic allocation solution, the \emph{Interweaved-Cliques (IC) design}, whose information-theoretic-inspired interweaved clique structure simultaneously achieves order-optimal communication and computation costs, for a large class of decompositions $\mathbf{X}$. This optimality is derived from our achievability and converse bounds, which reveal -- under reasonable assumptions on the density of $\mathbf{X}$ -- that the optimal scaling of the communication cost takes the form $n/N^{1/d}$, revealing that our design achieves the order-optimal \textit{partitioning gain} that scales as $N^{1/d}$, while also achieving an order-optimal computation cost. Interestingly, this order optimality is achieved in a deterministic manner, and very importantly, it is achieved blindly from $\mathbf{X}$, therefore enabling multiple desired functions to be computed without reshuffling files.

Universal and Asymptotically Optimal Data and Task Allocation in Distributed Computing

TL;DR

The paper addresses the joint optimization of communication and computation in a master–worker distributed setting where a function decomposes into -wise subfunctions over a library of files. It introduces the deterministic Interweaved-Cliques (IC) design, which partitions the -uniform hypergraph of dependencies into groups to achieve and keep bounded (typically ) with high probability, for a broad class of decompositions . The main theoretical contributions include a tight lower bound and an IC-based achievability showing order-optimal partitioning gain , plus corollaries that extend optimality across multiple decompositions and degrees. Practically, the IC design provides a universal, reshuffle-free data placement that remains valid across many function representations, enabling efficient multi-function computation in distributed systems with parallel communication channels.

Abstract

We study the joint minimization of communication and computation costs in distributed computing, where a master node coordinates workers to evaluate a function over a library of files. Assuming that the function is decomposed into an arbitrary subfunction set , with each subfunction depending on input files, renders our distributed computing problem into a -uniform hypergraph edge partitioning problem wherein the edge set (subfunction set), defined by -wise dependencies between vertices (files) must be partitioned across disjoint groups (workers). The aim is to design a file and subfunction allocation, corresponding to a partition of , that minimizes the communication cost , representing the maximum number of distinct files per server, while also minimizing the computation cost corresponding to a maximal worker subfunction load. For a broad range of parameters, we propose a deterministic allocation solution, the \emph{Interweaved-Cliques (IC) design}, whose information-theoretic-inspired interweaved clique structure simultaneously achieves order-optimal communication and computation costs, for a large class of decompositions . This optimality is derived from our achievability and converse bounds, which reveal -- under reasonable assumptions on the density of -- that the optimal scaling of the communication cost takes the form , revealing that our design achieves the order-optimal \textit{partitioning gain} that scales as , while also achieving an order-optimal computation cost. Interestingly, this order optimality is achieved in a deterministic manner, and very importantly, it is achieved blindly from , therefore enabling multiple desired functions to be computed without reshuffling files.
Paper Structure (28 sections, 12 theorems, 204 equations, 2 figures)

This paper contains 28 sections, 12 theorems, 204 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Brief Summary of Main Results
  3. Paper Organization and Notation
  4. Problem Statement and Setup
  5. Main Results
  6. Achievable Scheme: Interweaved Clique (IC) Design
  7. Case 1: $k=\frac{n}{s}$ for some $s\in \mathbb{Z}^+$
  8. Case 2: $k \nmid n$
  9. Extension of the Partition from $N'$ Groups to $N$ Groups
  10. Partition Refinement
  11. Comparison
  12. Conclusions
  13. Proof of Lemmas
  14. Proof of Lemma \ref{['lem:ub-lb']}
  15. Proof of Lemma \ref{['lem:lemma_pi_g=0']}
  16. ...and 13 more sections

Key Result

Theorem 1

For the distributed computing setting with $n$ files, $N$ workers, subfunction degree $d$, and subfunction set $\mathbf{X} \subseteq \mathbf{A}_{n,d}$ with normalized size $\varphi = |\mathbf{X}| / \binom{n}{d}$, the optimal communication cost $\pi^\star_{\mathbf{X}}$ satisfies Furthermore, if $\varphi \ge \varphi_{\mathrm{min}}$, $d \le \frac{n}{32}$ and $N \le(\frac{9}{10}\sqrt{\frac{n}{d}})^{

Figures (2)

  • Figure 1: Distributed computing model: The desired function $F$ admits a decomposition $F = \Psi(\{\zeta_\mathcal{T}(\mathcal{W}_\mathcal{T}):\mathcal{T}\in \mathbf{X},|\mathcal{T}|=d\})$ for some $\mathbf{X}\subseteq \mathbf{A}_{n,d}$. The set of files communicated to worker $b$ is denoted with $\mathcal{W}^{(b)}$, while $\pi_{\mathbf{X}}$ denotes the worst-case communication cost across the master--worker links under the assumption of parallel, equal-capacity links. Similarly, the indices ($d$-tuples) of the subfunctions assigned for computation to worker $b$ is $\mathbf{\Phi}_{b}$, while $\delta_{\mathbf{X}}$ denotes the computational delay normalized by the minimum possible computational delay, assuming homogeneous workers.
  • Figure 2: Overview of the logical structure of the proofs of the paper, highlighting the dependencies among lemmas and theorems.

Theorems & Definitions (30)

  • Example 1
  • Remark 1
  • Example 2
  • Remark 2
  • Theorem 1
  • proof
  • Remark 3
  • Remark 4
  • Corollary 1
  • proof
  • ...and 20 more