Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Latent-Space Constraints in Personalized Federated Learning

Sana Ayromlou, Fatemeh Tavakoli, D. B. Emerson

TL;DR

The paper addresses activity drift due to data heterogeneity in personalized federated learning by introducing adaptive latent-space constraints based on Maximum Mean Discrepancy, focusing on MK-MMD and MMD-D to constrain feature representations within Ditto and MR-MTL. By optimizing kernels online and applying the penalties to multiple latent spaces, the approach yields robust improvements across datasets with pronounced feature heterogeneity, often outperforming traditional weight-based Ditto. The study demonstrates that adaptive latent-space penalties capture drift aspects not addressed by weight penalties, with MMD-D providing particularly consistent gains and strong performance when combined with Ditto on several benchmarks. These findings support the practical use of adaptive, heterogeneity-aware drift constraints to enhance pFL in real-world heterogeneous settings.

Abstract

Federated learning (FL) is an effective and widely used approach to training deep learning models on decentralized datasets held by distinct clients. FL also strengthens both security and privacy protections for training data. Common challenges associated with statistical heterogeneity between distributed datasets have spurred significant interest in personalized FL (pFL) methods, where models combine aspects of global learning with local modeling specific to each client's unique characteristics. This work investigates the efficacy of theoretically supported, adaptive MMD measures in pFL, primarily focusing on the Ditto framework, a state-of-the-art technique for distributed data heterogeneity. The use of such measures significantly improves model performance across a variety of tasks, especially those with pronounced feature heterogeneity. Additional experiments demonstrate that such measures are directly applicable to other pFL techniques and yield similar improvements across a number of datasets. Finally, the results motivate the use of constraints tailored to the various kinds of heterogeneity expected in FL systems.

Adaptive Latent-Space Constraints in Personalized Federated Learning

TL;DR

The paper addresses activity drift due to data heterogeneity in personalized federated learning by introducing adaptive latent-space constraints based on Maximum Mean Discrepancy, focusing on MK-MMD and MMD-D to constrain feature representations within Ditto and MR-MTL. By optimizing kernels online and applying the penalties to multiple latent spaces, the approach yields robust improvements across datasets with pronounced feature heterogeneity, often outperforming traditional weight-based Ditto. The study demonstrates that adaptive latent-space penalties capture drift aspects not addressed by weight penalties, with MMD-D providing particularly consistent gains and strong performance when combined with Ditto on several benchmarks. These findings support the practical use of adaptive, heterogeneity-aware drift constraints to enhance pFL in real-world heterogeneous settings.

Abstract

Federated learning (FL) is an effective and widely used approach to training deep learning models on decentralized datasets held by distinct clients. FL also strengthens both security and privacy protections for training data. Common challenges associated with statistical heterogeneity between distributed datasets have spurred significant interest in personalized FL (pFL) methods, where models combine aspects of global learning with local modeling specific to each client's unique characteristics. This work investigates the efficacy of theoretically supported, adaptive MMD measures in pFL, primarily focusing on the Ditto framework, a state-of-the-art technique for distributed data heterogeneity. The use of such measures significantly improves model performance across a variety of tasks, especially those with pronounced feature heterogeneity. Additional experiments demonstrate that such measures are directly applicable to other pFL techniques and yield similar improvements across a number of datasets. Finally, the results motivate the use of constraints tailored to the various kinds of heterogeneity expected in FL systems.
Paper Structure (32 sections, 9 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 32 sections, 9 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Beyond Weights: Feature-Drift Constraints
  5. Maximum Mean Discrepancy
  6. Multi-Kernel MMD
  7. MMD-D
  8. Adaptive MMD Measures in FL
  9. Datasets
  10. Experimental Setting
  11. Communication, Privacy, and Computation Implications
  12. Results
  13. Interaction of the Ditto and MMD Constraints
  14. Ablation Studies
  15. Conclusions, Limitations, and Future Work
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Feature-drift constraints applied to three latent spaces of a model, where $(\mathbf{x}, \mathbf{y})$ represent a batch of labeled data. In Ditto, the frozen global model (top) is used to constrain the local model (bottom) during client-side training.
  • Figure 2: Results for CIFAR-10 and Fed-ISIC2019 when varying $\lambda$ around the optimal value. The best result is reported for MMD-based penalty weights, $\mu$, drawn from $\{0.01, 0.1, 1.0\}$.
  • Figure 3: Evolution of accuracy on the CIFAR-10 ($\alpha=0.1$) dataset (left) with an increasing number of constraints at various model depths for the MMD measures with and without the standard Ditto drift penalty. Accuracy for Synthetic datasets (right) using a varying number of fixed kernels with uniform weight compared to using the kernel optimization procedures of MK-MMD or MMD-D.