Distributed Linearly Separable Computation with Arbitrary Heterogeneous Data Assignment
Ziting Zhang, Kai Wan, Minquan Cheng, Shuo Shao, Giuseppe Caire
TL;DR
This paper addresses distributed linearly separable computation under arbitrary heterogeneous data assignment with a single master and ${N}$ workers, formulating a $(K,N,C)$ model aimed at maximizing the computable dimension ${K}_{c}$ for a given communication cost ${C}$. It develops universal converse and achievable schemes that apply to both integer and fractional communication costs, leveraging a data-assignment sparsity structure ${\mathcal Z}$ and parameters ${\alpha}$ and $t$ to bound ${K}_{c}$; the key bounds coincide in certain regimes. The proposed scheme uses left-null-space encoding and a factorization ${\bf F}={\bf F}_{1}{\bf F}_{2}$ to satisfy encodability and decodability, with high-probability full-rank decoding as the field size grows. The framework generalizes prior gradient-coding and coded-distributed computation results to ${K}_{c}\ge 1$ under arbitrary heterogeneity and provides fractional-cost extensions by partitioning each dataset, enabling flexible, universal performance guarantees for large-scale, heterogeneous systems.
Abstract
Distributed linearly separable computation is a fundamental problem in large-scale distributed systems, requiring the computation of linearly separable functions over different datasets across distributed workers. This paper studies a heterogeneous distributed linearly separable computation problem, including one master and N distributed workers. The linearly separable task function involves Kc linear combinations of K messages, where each message is a function of one dataset. Distinguished from the existing homogeneous settings that assume each worker holds the same number of datasets, where the data assignment is carefully designed and controlled by the data center (e.g., the cyclic assignment), we consider a more general setting with arbitrary heterogeneous data assignment across workers, where `arbitrary' means that the data assignment is given in advance and `heterogeneous' means that the workers may hold different numbers of datasets. Our objective is to characterize the fundamental tradeoff between the computable dimension of the task function and the communication cost under arbitrary heterogeneous data assignment. Under the constraint of integer communication costs, for arbitrary heterogeneous data assignment, we propose a universal computing scheme and a universal converse bound by characterizing the structure of data assignment, where they coincide under some parameter regimes. We then extend the proposed computing scheme and converse bound to the case of fractional communication costs.