Structured Codes for Distributed Matrix Multiplication

TL;DR

The paper develops a structured coding framework for distributed computation of bilinear functions, notably dot and matrix products, when inputs are two correlated sources. By injecting non-linear pre-processing at the master and coupling it with Körner–Marton linear encodings, the authors derive achievability and converse results that beat traditional Slepian–Wolf limits in large-field regimes and quantify compression gains. The main practical contribution is the StPolyDot family of codes, which blends structured source coding with polynomial coding to reduce master communication, storage, and decoding costs, while enabling chain-matrix multiplications and information-theoretic security. Numerical results corroborate substantial communication savings and favorable computation tradeoffs, making the approach appealing for master–worker distributed settings with memory-bounded workers and privacy constraints.

Abstract

Our work addresses the well-known open problem of distributed computing of bilinear functions of two correlated sources ${\bf A}$ and ${\bf B}$. In a setting with two nodes, with the first node having access to ${\bf A}$ and the second to ${\bf B}$, we establish bounds on the optimal sum-rate that allows a receiver to compute an important class of non-linear functions, and in particular bilinear functions, including dot products $\langle {\bf A},{\bf B}\rangle$, and general matrix products ${\bf A}^{\intercal}{\bf B}$ over finite fields. The bounds are tight, for large field sizes, for which case we can derive the exact fundamental performance limits for all problem dimensions and a large class of sources. Our achievability scheme involves the design of non-linear transformations of ${\bf A}$ and ${\bf B}$, which are carefully calibrated to work synergistically with the structured linear encoding scheme by Körner and Marton. The subsequent converse derived here, calibrates the Han-Kobayashi approach to yield a relatively tight converse on the sum rate. We also demonstrate unbounded compression gains over Slepian-Wolf coding, depending on the source correlations. In the end, our work derives fundamental limits for distributed computing of a crucial class of functions, succinctly capturing the computation structures and source correlations. Our findings are subsequently applied to the practical master-workers-receiver framework, where each of $N$ distributed workers has a bounded memory reflecting a bounded computational capability. By combining the above scheme with the polynomial code framework, we design novel structured polynomial codes for distributed matrix multiplication, and show that our codes can surpass the performance of the existing state of art, while also adapting these new codes to support chain matrix multiplications and information-theoretically secure computations.

Figures (10)

• Figure 1: Distributed computation of ${\bf \mathbfcal{D}}={\bf A}^{\intercal}{\bf B}$.
• Figure 2: Gain, $\eta$ from Corollary \ref{['cor:innerproduct_length_m_binary']}. The flat (yellow) line marks $\eta=1$.
• Figure 3: Rate comparisons for various source PMFs. (Left) Corollary \ref{['cor:innerproduct_length_m_binary']} for $m=2$. (Middle) $m=1$, where $(a, b)\sim {\rm DSBS}(p)$. (Right) $m=2$, where $(a_{i},\ b_{i})\sim {\rm DSBS}(p)$, for each $i=1,2$.
• Figure 4: Rate (in log scale) versus $p$ for distributed computing of (Left) symmetric matrices ${\bf A}^{\intercal}{\bf B}={\bf B}^{\intercal}{\bf A}$ via distributed multiplication of matrices ${\bf A}, {\bf B}\in \mathbb{F}_2^{m\times m}$ for different $m$, and (Right) square matrices via distributed matrix multiplication for different $m$ and $l=2$, where the joint source PMF is given in Example \ref{['ex:general_matrix_q3']}.
• Figure 5: The distributed matrix multiplication framework considered in the current work. The colorings for the input submatrices and the subfunctions are chosen in accordance: different shades of orange denote $p_{i}^{(1)}(x_i)\triangleq {\bf \tilde{A}}_{i2}+{\bf \tilde{B}}_{i1}$ and the submatrices ${\bf \tilde{A}}_{i2} \ , {\bf \tilde{B}}_{i1}$, shades of blue denote $p_{i}^{(2)}(x_i)\triangleq {\bf \tilde{A}}_{i1}+{\bf \tilde{B}}_{i2}$ and ${\bf \tilde{A}}_{i1} \ , {\bf \tilde{B}}_{i2}$, and green denotes the non-linear parity polynomial $p_{i}^{(3)}(x_i)\triangleq {\bf \tilde{A}}_{i2}^{\intercal}{\bf \tilde{A}}_{i1}+{\bf \tilde{B}}_{i1}^{\intercal}{\bf \tilde{B}}_{i2}$, which is linearly separable in terms of ${\bf \tilde{A}}_i$ and ${\bf \tilde{B}}_i$.

Theorems & Definitions (68)

• Proposition 1
• proof
• Corollary 1
• proof
• Corollary 2
• proof
• Proposition 2
• proof
• Proposition 3
• proof