Tessellated Distributed Computing

TL;DR

This work models multi-user distributed computing of real-valued linearly-decomposable functions as a fixed-support matrix factorization problem DE ≈ F, where D encodes server-function assignments and E encodes server contributions and user connections. It introduces tessellation-based and SVD-based fixed-support designs that decompose F into tiles, yielding lossless schemes with exact capacity in key regimes and, in lossy scenarios, asymptotic error bounds derived from the Marchenko-Pastur distribution. The main contributions include (i) a capacity formula for lossless reconstruction under disjoint supports, (ii) a tessellated construction linking fixed-support factorization with tiling and SVD, and (iii) a stochastic lossy analysis providing tight bounds on average reconstruction error via MP statistics. Collectively, the results quantify the rate-distortion-computation-communication tradeoffs in distributed computing and reveal a principled, tile-based approach to scalable, reliable computation across heterogeneous servers.

Abstract

The work considers the $N$-server distributed computing scenario with $K$ users requesting functions that are linearly-decomposable over an arbitrary basis of $L$ real (potentially non-linear) subfunctions. In our problem, the aim is for each user to receive their function outputs, allowing for reduced reconstruction error (distortion) $ε$, reduced computing cost ($γ$; the fraction of subfunctions each server must compute), and reduced communication cost ($δ$; the fraction of users each server must connect to). For any given set of $K$ requested functions -- which is here represented by a coefficient matrix $\mathbf {F} \in \mathbb{R}^{K \times L}$ -- our problem is made equivalent to the open problem of sparse matrix factorization that seeks -- for a given parameter $T$, representing the number of shots for each server -- to minimize the reconstruction distortion $\frac{1}{KL}\|\mathbf {F} - \mathbf{D}\mathbf{E}\|^2_{F}$ overall $δ$-sparse and $γ$-sparse matrices $\mathbf{D}\in \mathbb{R}^{K \times NT}$ and $\mathbf{E} \in \mathbb{R}^{NT \times L}$. With these matrices respectively defining which servers compute each subfunction, and which users connect to each server, we here design our $\mathbf{D},\mathbf{E}$ by designing tessellated-based and SVD-based fixed support matrix factorization methods that first split $\mathbf{F}$ into properly sized and carefully positioned submatrices, which we then approximate and then decompose into properly designed submatrices of $\mathbf{D}$ and $\mathbf{E}$.

Figures (15)

• Figure 1: The $K$-user, $N$-server, $T$-shot setting. Each server $n$ computes the subfunctions in $\mathcal{S}_n=\{f_{i_{n,1}}(.),f_{i_{n,2}}(.),\hdots , f_{i_{n,|\mathcal{S}_n|}}(.)\}$ and communicates to users in $\mathcal{T}_{n,t}$, under computational constraint $|\mathcal{S}_n|\leq \Gamma\leq L$ and communication constraint $|\mathcal{T}_n|\leq \Delta \leq K$, yielding a system with normalized constraints $\gamma = \frac{\Gamma}{L}, \delta = \frac{\Delta}{K}$ and with an error constraint $\epsilon = \frac{\mathcal{E}}{KL}$, where $\gamma,\delta, \epsilon\in[0,1]$.
• Figure 2: Corresponding to Example \ref{['single-shot-example-simple']}, this figure illustrates the partitioning of $\mathbf{F}$ into 4 tiles of size $(\Delta \times \Gamma) = (3\times 5)$, and also illustrates the sparse tiling of $\mathbf{D}$ and $\mathbf{E}$ with tiles $\mathbf{L}_j$ and $\mathbf{R}_j$ respectively, resulting in the full tiling of $\mathbf{F} = \mathbf{DE}$ which is covered by the four $\mathbf{S}_j = \mathbf{L}_j \mathbf{R}_j, j \in [4]$ (see Figure \ref{['ex1']}), guaranteeing sparsity $\delta =\gamma = \frac{1}{2}$ for $\mathbf{D}$ and $\mathbf{E}$ respectively, thus satisfying the per-server communication and computing constraints, while yielding lossless reconstruction of $\mathbf{F}$ and thus of the desired functions.
• Figure 3: A problem setting with the same $K=6,L=10, \Delta=3$ and $\mathcal{E}=0$ as in Example \ref{['ex1']}, but a smaller computation cost $\Gamma =2$ corresponding to $\gamma = 1/5$. The number of servers used now for zero-error function recovery increases from $12$ to $20$.
• Figure 4: Pertaining to Example \ref{['ex:exampleDelta2']} with $K = 6,L = 10,T=1,\Gamma = 5$ and an optimal number of $N=12$ servers, the new tessellation pattern allows for a reduced $\Delta =2$ reflecting a reduction from $\delta = 1/2$ to $\delta = 1/3$.
• Figure 5: On the right we see the optimal performance for $T\geq \mathrm{min}(\Delta,\Gamma)$, which contrasts the blue achievable region with the red provably non-achievable region. On the left, we illustrate for the simple single-shot case, the two optimal points $(\gamma = \frac{K}{N},\delta= \frac{1}{K})$ and $(\gamma=\frac{1}{L},\delta = \frac{L}{N})$, which are compared to the operating points $A=(\gamma=1,\delta=1/K)$ and $B= (\gamma = 1/L,\delta=1)$ of two conceivable baseline schemes. Point $A=(\frac{1}{L},1)$ is that of a baseline fully-centralized scheme where servers $n\in[K]$ must compute all subfunctions (the rest are assigned no functions), while point $B=(1,\frac{1}{K})$ corresponds to a fully-parallelized baseline scheme where each server only computes one subfunction output and sends it, by necessity, to all users. The two points correspond to the trivial decompositions $\mathbf{F}=[\mathbf{I}_{K} \ \ \mathbf{0}_{(K, K-N)}] \cdot [\mathbf{F}^{\intercal} \ \ \mathbf{0}_{(L, N-K)} ]^{\intercal}$ and $\mathbf{F} = [\mathbf{F} \ \ \mathbf{0}_{(K,N-L)}] \cdot [\mathbf{I}_{L}^{} \ \ \mathbf{0}^{}_{(L,N-L)}]$ respectively.

Theorems & Definitions (59)

• Theorem 1
• proof
• Definition 1
• Example 1
• Example 2
• Example 3
• Corollary 1
• proof
• Corollary 2
• proof