Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective

TL;DR

The paper studies how data heterogeneity across clients impacts generalization in one-round distributed learning, presenting CMI-type bounds extended to K clients and linking to rate-distortion theory. It derives lossy and Jensen-Shannon divergence-based generalization bounds, plus explicit heterogeneity-dependent bounds for distributed SVM, and validates these results experimentally. The findings suggest that greater cross-client dissimilarity can improve the generalization of the aggregated model under the proposed frameworks, offering new insights for federated and distributed learning with heterogeneous data. These results provide principled, information-theoretic guarantees for understanding and controlling generalization in heterogeneous distributed systems, with practical implications for DSVM and beyond.

Abstract

In this paper, we investigate the effect of data heterogeneity across clients on the performance of distributed learning systems, i.e., one-round Federated Learning, as measured by the associated generalization error. Specifically, $K$ clients have each $n$ training samples generated independently according to a possibly different data distribution, and their individually chosen models are aggregated by a central server. We study the effect of the discrepancy between the clients' data distributions on the generalization error of the aggregated model. First, we establish in-expectation and tail upper bounds on the generalization error in terms of the distributions. In part, the bounds extend the popular Conditional Mutual Information (CMI) bound, which was developed for the centralized learning setting, i.e., $K=1$, to the distributed learning setting with an arbitrary number of clients $K \geq 1$. Then, we connect with information-theoretic rate-distortion theory to derive possibly tighter \textit{lossy} versions of these bounds. Next, we apply our lossy bounds to study the effect of data heterogeneity across clients on the generalization error for the distributed classification problem in which each client uses Support Vector Machines (DSVM). In this case, we establish explicit generalization error bounds that depend explicitly on the data heterogeneity degree. It is shown that the bound gets smaller as the degree of data heterogeneity across clients increases, thereby suggesting that DSVM generalizes better when the dissimilarity between the clients' training samples is bigger. This finding, which goes beyond DSVM, is validated experimentally through several experiments.

Figures (13)

• Figure 1: Studied distributed learning problem
• Figure 2: Illustration of (training) data generation for an example D-SVM problem with $K=2$ clients.
• Figure 3: Evolution of the exact generalization bounds of Theorem \ref{['SVM:K=2:het-hom']} (with the constants of the $\mathcal{O}$ approximation substituted as indicated in \ref{['eq:svm_pr_rate_def_9']}), as well as their ratio, as functions of the ball radius $\rho$, for both heterogeneous and homogeneous data settings. Parameters: $n = 1000$, $\theta = 1$, $K = 2$, $a_1 = (0.2, \mathbf{0}_{d-1})$, and $a_2 = (0.6, \mathbf{0}_{d-1})$.
• Figure 4: Evolution of the generalization bound \ref{['bound-special-case-DSVM-general-case']} for various degrees of data heterogeneity across clients.
• Figure 5: Evolution of the generalization bound \ref{['bound-special-case-Gaussian-DSVM-general-case']} for various degrees of data heterogeneity across clients.

Theorems & Definitions (13)

• Definition 1: Symmetric Priors
• Theorem 1
• Theorem 2
• Theorem 3
• Lemma 1
• Theorem 4
• Lemma 2
• Theorem 5
• Theorem 6
• Theorem 7