Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Language Model Architectures for Differentially Private Federated Learning

Jae Hun Ro, Srinadh Bhojanapalli, Zheng Xu, Yanxiang Zhang, Ananda Theertha Suresh

TL;DR

A scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network is proposed by modifying the sigmoid and tanh activations in the recurrent cell and it is shown that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments.

Abstract

Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the standard client optimizer for on device training in cross-device FL, favored for its memory and computational efficiency. However, in centralized training of neural language models, adaptive optimizers are preferred as they offer improved stability and performance. In light of this, we ask if language models can be modified such that they can be efficiently trained with SGD client optimizers and answer this affirmatively. We propose a scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network by modifying the sigmoid and tanh activations in the recurrent cell and show that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments. We further show that the proposed scale invariant modification also helps in federated learning of larger transformer models. Finally, we demonstrate the scale invariant modification is also compatible with other non-adaptive algorithms. Particularly, our results suggest an improved privacy utility trade-off in federated learning with differential privacy.

Efficient Language Model Architectures for Differentially Private Federated Learning

TL;DR

A scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network is proposed by modifying the sigmoid and tanh activations in the recurrent cell and it is shown that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments.

Abstract

Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the standard client optimizer for on device training in cross-device FL, favored for its memory and computational efficiency. However, in centralized training of neural language models, adaptive optimizers are preferred as they offer improved stability and performance. In light of this, we ask if language models can be modified such that they can be efficiently trained with SGD client optimizers and answer this affirmatively. We propose a scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network by modifying the sigmoid and tanh activations in the recurrent cell and show that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments. We further show that the proposed scale invariant modification also helps in federated learning of larger transformer models. Finally, we demonstrate the scale invariant modification is also compatible with other non-adaptive algorithms. Particularly, our results suggest an improved privacy utility trade-off in federated learning with differential privacy.
Paper Structure (13 sections, 1 theorem, 7 equations, 3 figures, 4 tables)

This paper contains 13 sections, 1 theorem, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Scale Invariant Architectures
  4. Previous scale invariant architectures
  5. New scale invariant architectures
  6. Experiments with Federated learning
  7. Federated experiments on public datasets
  8. Live production experiments
  9. Experiments with differentially private federated learning
  10. Conclusion
  11. Federated experiments on public datasets details
  12. Live production experiment details
  13. Experiments with differentially private federated learning details

Key Result

Proposition 1

Both $\text{SI-}\sigma$ and $\text{SI-}\tanh$ are scale invariant functions.

Figures (3)

  • Figure 1: Perplexity and accuracy on the Stack Overflow test dataset with shading indicating standard deviation over $5$ random seeds.
  • Figure 2: Perplexity and accuracy from live experiments on English virtual keyboard devices training from randomly initialized model weights.
  • Figure 3: Smoothed perplexity and in-vocab-accuracy from DP live experiments on English virtual keyboard devices.

Theorems & Definitions (2)

  • Proposition 1
  • proof