Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning

Yuchang Sun, Marios Kountouris, Jun Zhang

TL;DR

This work addresses maximizing local generalization in cross-silo FL under data heterogeneity by deriving a client-specific generalization bound and formulating a gradient-distance-based utility to guide collaboration. It introduces HCCT, a hierarchical clustering-based training scheme that adaptively merges clients into groups to improve overall utility without pre-specifying the number of groups, and proves convergence for non-convex losses. Empirical results across digit, FMNIST, and CIFAR-10 tasks show HCCT can outperform baselines by tailoring collaboration to data similarity and sample size, while HCCT-P demonstrates added gains when personalization is combined with grouping. The approach offers a practical, server-driven mechanism to balance data sharing and heterogeneity, with implications for scalable, privacy-preserving multi-client learning in realistic, non-IID settings.

Abstract

Federated learning (FL) has attracted vivid attention as a privacy-preserving distributed learning framework. In this work, we focus on cross-silo FL, where clients become the model owners after training and are only concerned about the model's generalization performance on their local data. Due to the data heterogeneity issue, asking all the clients to join a single FL training process may result in model performance degradation. To investigate the effectiveness of collaboration, we first derive a generalization bound for each client when collaborating with others or when training independently. We show that the generalization performance of a client can be improved only by collaborating with other clients that have more training data and similar data distribution. Our analysis allows us to formulate a client utility maximization problem by partitioning clients into multiple collaborating groups. A hierarchical clustering-based collaborative training (HCCT) scheme is then proposed, which does not need to fix in advance the number of groups. We further analyze the convergence of HCCT for general non-convex loss functions which unveils the effect of data similarity among clients. Extensive simulations show that HCCT achieves better generalization performance than baseline schemes, whereas it degenerates to independent training and conventional FL in specific scenarios.

How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning

TL;DR

This work addresses maximizing local generalization in cross-silo FL under data heterogeneity by deriving a client-specific generalization bound and formulating a gradient-distance-based utility to guide collaboration. It introduces HCCT, a hierarchical clustering-based training scheme that adaptively merges clients into groups to improve overall utility without pre-specifying the number of groups, and proves convergence for non-convex losses. Empirical results across digit, FMNIST, and CIFAR-10 tasks show HCCT can outperform baselines by tailoring collaboration to data similarity and sample size, while HCCT-P demonstrates added gains when personalization is combined with grouping. The approach offers a practical, server-driven mechanism to balance data sharing and heterogeneity, with implications for scalable, privacy-preserving multi-client learning in realistic, non-IID settings.

Abstract

Federated learning (FL) has attracted vivid attention as a privacy-preserving distributed learning framework. In this work, we focus on cross-silo FL, where clients become the model owners after training and are only concerned about the model's generalization performance on their local data. Due to the data heterogeneity issue, asking all the clients to join a single FL training process may result in model performance degradation. To investigate the effectiveness of collaboration, we first derive a generalization bound for each client when collaborating with others or when training independently. We show that the generalization performance of a client can be improved only by collaborating with other clients that have more training data and similar data distribution. Our analysis allows us to formulate a client utility maximization problem by partitioning clients into multiple collaborating groups. A hierarchical clustering-based collaborative training (HCCT) scheme is then proposed, which does not need to fix in advance the number of groups. We further analyze the convergence of HCCT for general non-convex loss functions which unveils the effect of data similarity among clients. Extensive simulations show that HCCT achieves better generalization performance than baseline schemes, whereas it degenerates to independent training and conventional FL in specific scenarios.
Paper Structure (22 sections, 4 theorems, 43 equations, 6 figures, 5 tables, 2 algorithms)

This paper contains 22 sections, 4 theorems, 43 equations, 6 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Cross-Silo FL
  5. Motivating Example
  6. Problem Formulation
  7. Theoretical Analysis of Generalization Performance
  8. Client Utility Maximization
  9. Hierarchical Clustering-based Collaborative Training Scheme
  10. Discussions
  11. Efficient Estimation of Client Similarity
  12. Convergence Analysis
  13. Simulation Results
  14. Setup
  15. Performance Comparison
  16. ...and 7 more sections

Key Result

Theorem 1

Consider the loss function with $\mu$-strong convexity and $L$-Lipschitz continuity. For a prediction function $h$ and any $0 <\delta < 1$, the following holds: with probability $1-\delta$, where $d_1\left(\hat{\mathcal{P}}_{G_i}, \mathcal{P}_{i}\right)$ is the distribution divergence (e.g., $\mathcal{H}$-divergence) between $\hat{\mathcal{P}}_{G_i}$ and $\mathcal{P}_{i}$, and $\lambda = \min \le

Figures (6)

  • Figure 1: An example of collaboration pattern among three clients in cross-silo FL.
  • Figure 2: An illustration of the client partition process in HCCT with six clients, where the same shapes indicate similar data distributions. In the beginning, each client is a singleton group. In Step (1), clients $1$ and $2$ are clustered together since $(1,2)=\arg\max_{(k_1,k_2)} B(k_1,k_2)$. In Step (2), clients $4$ and $5$ are clustered together since $(4,5)=\arg\max_{(k_1,k_2)} B(k_1,k_2)$. In Step (3), group $1$ and client $3$ are clustered together since $(1,3)=\arg\max_{(k_1,k_2)} B(k_1,k_2)$. In Step (4), we stop partitioning since there is no benefit, i.e., $B(k_1, k_2)\leq 0, \forall k_1\neq k_2$. The final group number in this example is $3$.
  • Figure 3: Local test error (mean) vs. training iterations in the CIFAR-10 dataset with $N_{new}$ new clients.
  • Figure 4: Local test error (mean) in the CIFAR-10 dataset with different values of $\alpha$.
  • Figure 5: Local test error (mean) and utility in the Digit dataset with different numbers of groups.
  • ...and 1 more figures

Theorems & Definitions (9)

  • Theorem 1
  • Proof
  • Remark 1
  • Theorem 2
  • proof
  • Corollary 1
  • proof
  • Lemma 1
  • proof