Order Optimal Cascaded Code Distributed Computing With Low Complexity and Improved Flexibility
Mingming Zhang, Youlong Wu, Minquan Cheng, Dianhua Wu
TL;DR
This paper tackles the MapReduce-level communication bottleneck in cascaded coded distributed computing by introducing a grouping-based, low-complexity scheme operable under K/s ∈ ℕ. It achieves two-round Shuffle delivery with multicast gains close to r+s−1 and (r+s−1)(1−1/s), while dramatically reducing the required numbers of input files N and output functions Q, and enabling one-shot decoding over the binary field. A new information-theoretic converse is derived, and the scheme is shown to be order-optimal within a factor of 2 (asymptotically optimal as K grows for fixed r). Compared to prior flexible-K schemes, this approach simultaneously lowers N and Q requirements and improves communication efficiency, offering practical gains for large-scale distributed computing systems.
Abstract
Coded distributed computing (CDC), proposed by Li \emph{et al.}, offers significant potential for reducing the communication load in MapReduce computing systems. In cascaded CDC with $K$ nodes, $N$ input files, and $Q$ output functions, each input file will be mapped by $r\geq 1$ nodes and each output function will be computed by $s>1$ nodes such that coding techniques can be applied to generate multicast opportunities. However, a significant limitation of most existing coded distributed computing schemes is their requirement to split the original data into a large number of input files (or output functions) that grows exponentially with $K$, which significantly increases the coding complexity and degrades the system performance. In this paper, we focus on the case of $K/s\in\mathbb{N}$, deliberately designing the strategy of data placement and output functions assignment, such that a low-complexity CDC scheme is achievable. The main advantages of the proposed scheme include: 1) the multicast gains equal to $(r+s-1)(1-1/s)$ and $r+s-1$ which is approximately $r+s-1$ when $s$ is relatively large, and the communication load potentially better than the well-known scheme proposed by Li \emph{et al.}; 2) the proposed scheme requires significantly less input files and output functions; 3) all the operations are implemented over the binary field $\mathbb{F}_2$ with the one-shot fashion (i.e., each node can decode its requested content immediately upon receiving the multicast message during the current time slot). Finally, we derive a new information-theoretic converse bound for the cascaded CDC framework under the proposed strategies of data placement and output functions assignment. We demonstrate that the communication load of the proposed scheme is order optimal within a factor of $2$; and is also approximately optimal when $K$ is sufficiently large for a given $r$.