Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedLog: Personalized Federated Classification with Less Communication and More Flexibility

Haolin Yu, Guojun Zhang, Pascal Poupart

TL;DR

This work proposes to share sufficient data summaries instead of raw model parameters to reduce the overhead of FRL algorithms, and extends it with differential privacy framework to ensure formal privacy guarantee.

Abstract

Federated representation learning (FRL) aims to learn personalized federated models with effective feature extraction from local data. FRL algorithms that share the majority of the model parameters face significant challenges with huge communication overhead. This overhead stems from the millions of neural network parameters and slow aggregation progress of the averaging heuristic. To reduce the overhead, we propose to share sufficient data summaries instead of raw model parameters. The data summaries encode minimal sufficient statistics of an exponential family, and Bayesian inference is utilized for global aggregation. It helps to reduce message sizes and communication frequency. To further ensure formal privacy guarantee, we extend it with differential privacy framework. Empirical results demonstrate high learning accuracy with low communication overhead of our method.

FedLog: Personalized Federated Classification with Less Communication and More Flexibility

TL;DR

This work proposes to share sufficient data summaries instead of raw model parameters to reduce the overhead of FRL algorithms, and extends it with differential privacy framework to ensure formal privacy guarantee.

Abstract

Federated representation learning (FRL) aims to learn personalized federated models with effective feature extraction from local data. FRL algorithms that share the majority of the model parameters face significant challenges with huge communication overhead. This overhead stems from the millions of neural network parameters and slow aggregation progress of the averaging heuristic. To reduce the overhead, we propose to share sufficient data summaries instead of raw model parameters. The data summaries encode minimal sufficient statistics of an exponential family, and Bayesian inference is utilized for global aggregation. It helps to reduce message sizes and communication frequency. To further ensure formal privacy guarantee, we extend it with differential privacy framework. Empirical results demonstrate high learning accuracy with low communication overhead of our method.
Paper Structure (31 sections, 9 theorems, 17 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 31 sections, 9 theorems, 17 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Federated Representation Learning
  4. Method
  5. Exponential Family
  6. Model Construction
  7. Notation.
  8. FedLog
  9. Interpretation and FedLog-C
  10. Differential Privacy
  11. Experiments
  12. Synthetic
  13. Communication Cost
  14. Celeba
  15. Flexible Architecture
  16. ...and 16 more sections

Key Result

Theorem 2

If $({\boldsymbol{\phi}_1}, y_1), ({\boldsymbol{\phi}_2}, y_2), \cdots, ({\boldsymbol{\phi}_n}, y_n)$ are i.i.d. samples from the exponential family defined with Eq. likeli, then ${\bf T}(({\boldsymbol{\phi}_1}, y_1), \cdots, ({\boldsymbol{\phi}_n}, y_n)) = \sum_{i=1}^n {\boldsymbol{\phi}_i}\otimes{

Figures (2)

  • Figure 1: Synthetic experiments. Dots are data points or local representations: dark green: client 0 class 0; light green: client 1 class 0; Dark blue: client 0 class 1; light blue: client 1 class 1. Dashed lines are linear separators. Accuracy results are averaged over 6 seeds.
  • Figure 2: From top to bottom: Celeba, flexible architecture, and differential privacy results. Colored area shows mean $\pm$ standard deviation of the accuracy.

Theorems & Definitions (21)

  • Definition 1
  • Theorem 2
  • proof
  • Definition 3
  • Theorem 4
  • proof
  • Definition A.5: 6.2.1 in casella2024statistical
  • Theorem A.6: 6.2.10 in casella2024statistical
  • Definition A.7: 6.2.11 in casella2024statistical
  • Theorem A.8: 6.2.13 in casella2024statistical
  • ...and 11 more