Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Learning With L0 Constraint Via Probabilistic Gates For Sparsity

Krishna Harsha Kovelakuntla Huthasana, Alireza Olama, Andreas Lundell

TL;DR

This work tackles sparsity in federated learning by imposing an $L_{0}$ density constraint on global parameters and enforcing it through a reparameterization with probabilistic gates. The authors derive the constraint from entropy maximization of stochastic gates, leading to a Gibbs-Boltzmann formulation and a tractable variational objective that supports gradient-based optimization. They introduce FLoPS and its communication-efficient variant FLoPS-PA, enabling synchronous distributed learning with test-time sparsity and reduced uplink/downlink traffic while preserving statistical performance on linear and nonlinear models. Empirical results on synthetic data and real datasets (RCV1, MNIST, EMNIST) demonstrate effective sparsity control under data/client heterogeneity and show superior or competitive accuracy with substantially improved communication efficiency.

Abstract

Federated Learning (FL) is a distributed machine learning setting that requires multiple clients to collaborate on training a model while maintaining data privacy. The unaddressed inherent sparsity in data and models often results in overly dense models and poor generalizability under data and client participation heterogeneity. We propose FL with an L0 constraint on the density of non-zero parameters, achieved through a reparameterization using probabilistic gates and their continuous relaxation: originally proposed for sparsity in centralized machine learning. We show that the objective for L0 constrained stochastic minimization naturally arises from an entropy maximization problem of the stochastic gates and propose an algorithm based on federated stochastic gradient descent for distributed learning. We demonstrate that the target density (rho) of parameters can be achieved in FL, under data and client participation heterogeneity, with minimal loss in statistical performance for linear and non-linear models: Linear regression (LR), Logistic regression (LG), Softmax multi-class classification (MC), Multi-label classification with logistic units (MLC), Convolution Neural Network (CNN) for multi-class classification (MC). We compare the results with a magnitude pruning-based thresholding algorithm for sparsity in FL. Experiments on synthetic data with target density down to rho = 0.05 and publicly available RCV1, MNIST, and EMNIST datasets with target density down to rho = 0.005 demonstrate that our approach is communication-efficient and consistently better in statistical performance.

Federated Learning With L0 Constraint Via Probabilistic Gates For Sparsity

TL;DR

This work tackles sparsity in federated learning by imposing an density constraint on global parameters and enforcing it through a reparameterization with probabilistic gates. The authors derive the constraint from entropy maximization of stochastic gates, leading to a Gibbs-Boltzmann formulation and a tractable variational objective that supports gradient-based optimization. They introduce FLoPS and its communication-efficient variant FLoPS-PA, enabling synchronous distributed learning with test-time sparsity and reduced uplink/downlink traffic while preserving statistical performance on linear and nonlinear models. Empirical results on synthetic data and real datasets (RCV1, MNIST, EMNIST) demonstrate effective sparsity control under data/client heterogeneity and show superior or competitive accuracy with substantially improved communication efficiency.

Abstract

Federated Learning (FL) is a distributed machine learning setting that requires multiple clients to collaborate on training a model while maintaining data privacy. The unaddressed inherent sparsity in data and models often results in overly dense models and poor generalizability under data and client participation heterogeneity. We propose FL with an L0 constraint on the density of non-zero parameters, achieved through a reparameterization using probabilistic gates and their continuous relaxation: originally proposed for sparsity in centralized machine learning. We show that the objective for L0 constrained stochastic minimization naturally arises from an entropy maximization problem of the stochastic gates and propose an algorithm based on federated stochastic gradient descent for distributed learning. We demonstrate that the target density (rho) of parameters can be achieved in FL, under data and client participation heterogeneity, with minimal loss in statistical performance for linear and non-linear models: Linear regression (LR), Logistic regression (LG), Softmax multi-class classification (MC), Multi-label classification with logistic units (MLC), Convolution Neural Network (CNN) for multi-class classification (MC). We compare the results with a magnitude pruning-based thresholding algorithm for sparsity in FL. Experiments on synthetic data with target density down to rho = 0.05 and publicly available RCV1, MNIST, and EMNIST datasets with target density down to rho = 0.005 demonstrate that our approach is communication-efficient and consistently better in statistical performance.
Paper Structure (11 sections, 31 equations, 7 figures, 2 tables, 2 algorithms)

This paper contains 11 sections, 31 equations, 7 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. L0 constrained formulation
  3. Centralized ML
  4. Entropy Maximization of Stochastic Gates
  5. FL with Reparameterization
  6. Distributed Optimization Algorithm
  7. Experiments
  8. Experiments on synthetic data sets
  9. Experiments on real datasets
  10. Conclusion
  11. Appendix

Figures (7)

  • Figure 1: The figure corresponds to results for synthetic data generated using a signal-to-noise (SNR) ratio of 20 and a covariance matrix generated using a 0.2 correlation factor. Here, (a) to (c) correspond to the expected gates of FLoPS: achieving $5\%$ target density of gates during training in heterogeneous conditions of data and client participation (HTC) in LR, LG, and MC cases, respectively. The blue dotted line corresponds to the round at which $\log \alpha$ scaling starts, corresponding to the target density.
  • Figure 2: The figure corresponds to results for synthetic data generated using a signal-to-noise (SNR) ratio of 20 and a covariance matrix generated using a 0.2 correlation factor. Here, (a) and (b) correspond to the soft IOU heat map for test time gates in FLoPS for $5\%$ target density of gates and IOU heat map of binary masks in FedIter-HT for the same level target density during training in heterogeneous conditions of data and client participation (HTC) in the LR case, respectively.
  • Figure 3: The figure corresponds to test $R^2$ of FLoPS-PA for various weights $\eta_{\tilde{\theta}}$ and gates $\eta_{\phi}$ learning rates while keeping the other constant for LR with $5\%$ target density on synthetic data under data and client participation heterogeneity.
  • Figure 4: The figure corresponds to results of FLoPS-PA and FedIter-HT with $5\%$ target density on synthetic LR and LG data, along with test $R^2$ in multi-run experiments with LR.
  • Figure 5: The figure corresponds to results of FLoPS-PA, FedIter-HT, and Central FLoPS-PA with on RCV1 at target density of $0.005\%$, MNIST and EMNIST data at target density of $5\%$ at heterogeneous conditions of data and client participation.
  • ...and 2 more figures