Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wireless MapReduce Arrays for Coded Distributed Computing

Elizabath Peter, K. K. Krishnan Namboodiri, B. Sundar Rajan

TL;DR

This work presents schemes that require the number of files to be in the order of the number of nodes and achieve the same performance as the existing scheme, and designs a structure called wireless MapReduce array that succinctly represents all three phases in a single array.

Abstract

We consider a wireless distributed computing system based on the MapReduce framework, which consists of three phases: \textit{Map}, \textit{Shuffle}, and \textit{Reduce}. The system consists of a set of distributed nodes assigned to compute arbitrary output functions depending on a file library. The computation of the output functions is decomposed into Map and Reduce functions, and the Shuffle phase, which involves the data exchange, links the two. In our model, the Shuffle phase communication happens over a full-duplex wireless interference channel. For this setting, a coded wireless MapReduce distributed computing scheme exists in the literature, achieving optimal performance under one-shot linear schemes. However, the scheme requires the number of input files to be very large, growing exponentially with the number of nodes. We present schemes that require the number of files to be in the order of the number of nodes and achieve the same performance as the existing scheme. The schemes are obtained by designing a structure called wireless MapReduce array that succinctly represents all three phases in a single array. The wireless MapReduce arrays can also be obtained from the extended placement delivery arrays known for multi-antenna coded caching schemes.

Wireless MapReduce Arrays for Coded Distributed Computing

TL;DR

This work presents schemes that require the number of files to be in the order of the number of nodes and achieve the same performance as the existing scheme, and designs a structure called wireless MapReduce array that succinctly represents all three phases in a single array.

Abstract

We consider a wireless distributed computing system based on the MapReduce framework, which consists of three phases: \textit{Map}, \textit{Shuffle}, and \textit{Reduce}. The system consists of a set of distributed nodes assigned to compute arbitrary output functions depending on a file library. The computation of the output functions is decomposed into Map and Reduce functions, and the Shuffle phase, which involves the data exchange, links the two. In our model, the Shuffle phase communication happens over a full-duplex wireless interference channel. For this setting, a coded wireless MapReduce distributed computing scheme exists in the literature, achieving optimal performance under one-shot linear schemes. However, the scheme requires the number of input files to be very large, growing exponentially with the number of nodes. We present schemes that require the number of files to be in the order of the number of nodes and achieve the same performance as the existing scheme. The schemes are obtained by designing a structure called wireless MapReduce array that succinctly represents all three phases in a single array. The wireless MapReduce arrays can also be obtained from the extended placement delivery arrays known for multi-antenna coded caching schemes.
Paper Structure (9 sections, 4 theorems, 14 equations)

This paper contains 9 sections, 4 theorems, 14 equations.

Table of Contents

  1. Introduction
  2. System Model
  3. Map Phase
  4. Shuffle Phase
  5. Reduce Phase
  6. Main Results
  7. Constructions of Wireless MapReduce Arrays
  8. Wireless MapReduce Arrays from Extended Placement Delivery Arrays (EPDAs) in NPRYWCQC
  9. Conclusion

Key Result

Lemma 1

Given a $(K,N,r,S)$ wireless MapReduce array, it is possible to obtain a wireless MapReduce scheme for a system having $K$ users, $Q=K$ output functions, and $N$ files with a computation load $r$ and a normalized delivery time $L = S/NQ= \frac{1}{\min\{2r,K\}}\left(1-\frac{r}{K} \right)$.

Theorems & Definitions (9)

  • Definition 1: Wireless MapReduce Array
  • Example 1
  • Lemma 1
  • Theorem 1
  • proof
  • Example 2
  • Example 3
  • Lemma 2
  • Corollary 1