Concentration Inequalities for Exchangeable Tensors and Matrix-valued Data

TL;DR

This paper develops Hoeffding and Bernstein concentration bounds for structured, exchangeable data in two rich settings: mode‑exchangeable tensors and sequences of matrix‑valued data. By introducing mode symmetry and, when possible, commutativity pairs, it yields sharp bounds with optimal constants and clarifies how concentration degrades relative to the i.i.d. case as dimensionality grows. The theoretical contributions subsume and sharpen previous results for weighted sums, i.i.d. matrix inequalities, and combinatorial matrix sums, while enabling applications to multi‑factor experimental design, fixed‑design sketching, and federated averaging with sketching. The numerical experiments corroborate the bounds and demonstrate practical benefits of the proposed structured‑sampling and sketching schemes in reducing variance and improving estimation accuracy.

Abstract

We study concentration inequalities for structured weighted sums of random data, including (i) tensor inner products and (ii) sequential matrix sums. We are interested in tail bounds and concentration inequalities for those structured weighted sums under exchangeability, extending beyond the classical framework of independent terms. We develop Hoeffding and Bernstein bounds provided with structure-dependent exchangeability. Along the way, we recover known results in weighted sum of exchangeable random variables and i.i.d. sums of random matrices to the optimal constants. Notably, we develop a sharper concentration bound for combinatorial sum of matrix arrays than the results previously derived from Chatterjee's method of exchangeable pairs. For applications, the richer structures provide us with novel analytical tools for estimating the average effect of multi-factor response models and studying fixed-design sketching methods in federated averaging. We apply our results to these problems, and find that our theoretical predictions are corroborated by numerical evidence.

Figures (3)

• Figure 1: Numerical simulations of the sub-sampling procedures. Under both $(N_1, N_2, N_3)$ configurations, we simulate Cartesian product sub-sampling with (i) $|I|=0.4N_1, |J|=N_2, |K|=N_3$; (ii) $|I|=N_1, |J|=0.4N_2, |K|=N_3$; (iii) $|I|=N_1, |J|=N_2, |K|=0.4N_3$; and also include (iv) i.i.d. sub-sampling with $\mathsf{Bernoulli}(0.4)$. For each $N$ we run $T=1,000$ independent trials for each of the procedure, and report the mean of the absolute estimation errors of the Horvitz-Thompson estimator $|\widehat{\mu} - \mu|$, with error bands between the 25th and 75th quantiles.
• Figure 2: Violin plots of the error statistic $\|\widehat{\theta} - \overline{\theta}\|$ when $\rho = 0.99$, comparing the sketching schemes: (A) fixed-design DST, (B) sub-sampled DST (without replacement), (C) sub-sampled DST (with replacement) and (D) Gaussian sketching. We run the experiments for $q' \in \{200, 500, 800, 1000\}$ over $100$ trials. The bars "–" represent the medians and dots "·" represent the means. The curves connect the means between different values of $q'$.
• Figure 3: Numerical simulations of sketching RTFA for (i) blue: Fixed design DST sketching vs. (ii) orange: Gaussian sketching. We run the experiments for $\sigma^2 = 0.2$, $\lambda = 0.5$ and $q' \in \{200, 500, 800, 1000\}$ over $100$ trials. We plot the mean error curves of $\|\overline{\theta}_t - \overline{\theta}^\star\|$ at each iteration $t$ with error bands between the 25th and 75th quantiles.

Theorems & Definitions (27)

• Definition 2.1: Mode exchangeability
• Theorem 1: Hoeffding-type MGF bound under mode-exchangeability
• Theorem 2: Bernstein-type MGF bound under mode-exchangeability
• Corollary 2.1
• Theorem 3: Generic Hoeffding-type bound for exchangeable matrix-valued data
• Remark 1
• Theorem 4: Hoeffding-type MGF bound for exchangeable matrix-valued data
• Theorem 5: Bernstein-type MGF bound for exchangeable matrix-valued data
• Corollary 4.1: Scalar weighted sum of exchangeable r.v.s
• Corollary 4.2: Matrix weighted sum of exchangeable r.v.s