Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient and Privacy-Preserving Decentralized Meta-Learning

Hansi Yang, James T. Kwok

TL;DR

This work tackles decentralized meta-learning across heterogeneous tasks under communication and privacy constraints. It introduces LoDMeta, which uses local auxiliary parameters and Gaussian perturbations within a random-walk decentralized framework to achieve low communication cost while providing network DP guarantees. Theoretical results show the method preserves the same asymptotic convergence rate as centralized meta-learning, and empirical studies on mini-ImageNet and Meta-Dataset validate competitive accuracy with reduced data sharing and privacy protection. Overall, LoDMeta enables scalable, privacy-preserving meta-learning without a central server, applicable to distributed, data-limited environments.

Abstract

Distributed learning, which does not require gathering training data in a central location, has become increasingly important in the big-data era. In particular, random-walk-based decentralized algorithms are flexible in that they do not need a central server trusted by all clients and do not require all clients to be active in all iterations. However, existing distributed learning algorithms assume that all learning clients share the same task. In this paper, we consider the more difficult meta-learning setting, in which different clients perform different (but related) tasks with limited training data. To reduce communication cost and allow better privacy protection, we propose LoDMeta (Local Decentralized Meta-learning) with the use of local auxiliary optimization parameters and random perturbations on the model parameter. Theoretical results are provided on both convergence and privacy analysis. Empirical results on a number of few-shot learning data sets demonstrate that LoDMeta has similar meta-learning accuracy as centralized meta-learning algorithms, but does not require gathering data from each client and is able to better protect data privacy for each client.

Communication-Efficient and Privacy-Preserving Decentralized Meta-Learning

TL;DR

This work tackles decentralized meta-learning across heterogeneous tasks under communication and privacy constraints. It introduces LoDMeta, which uses local auxiliary parameters and Gaussian perturbations within a random-walk decentralized framework to achieve low communication cost while providing network DP guarantees. Theoretical results show the method preserves the same asymptotic convergence rate as centralized meta-learning, and empirical studies on mini-ImageNet and Meta-Dataset validate competitive accuracy with reduced data sharing and privacy protection. Overall, LoDMeta enables scalable, privacy-preserving meta-learning without a central server, applicable to distributed, data-limited environments.

Abstract

Distributed learning, which does not require gathering training data in a central location, has become increasingly important in the big-data era. In particular, random-walk-based decentralized algorithms are flexible in that they do not need a central server trusted by all clients and do not require all clients to be active in all iterations. However, existing distributed learning algorithms assume that all learning clients share the same task. In this paper, we consider the more difficult meta-learning setting, in which different clients perform different (but related) tasks with limited training data. To reduce communication cost and allow better privacy protection, we propose LoDMeta (Local Decentralized Meta-learning) with the use of local auxiliary optimization parameters and random perturbations on the model parameter. Theoretical results are provided on both convergence and privacy analysis. Empirical results on a number of few-shot learning data sets demonstrate that LoDMeta has similar meta-learning accuracy as centralized meta-learning algorithms, but does not require gathering data from each client and is able to better protect data privacy for each client.
Paper Structure (25 sections, 10 theorems, 63 equations, 9 figures, 5 tables, 3 algorithms)

This paper contains 25 sections, 10 theorems, 63 equations, 9 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Random-Walk Decentralized Optimization
  4. Privacy in Distributed Learning
  5. Proposed Method
  6. Problem Formulation
  7. Reducing Communication Cost
  8. Protecting Privacy
  9. Theoretical Analysis
  10. Analysis on Convergence Rate and Communication Cost
  11. Privacy Analysis
  12. Experiments
  13. Setup
  14. Results
  15. Conclusion
  16. ...and 10 more sections

Key Result

Proposition 4.2

For any ${\bm{u}}, {\bm{w}} \in\mathbb{R}^d$, we have $\|\nabla \mathcal{L}({\bm{u}}) - \nabla \mathcal{L}({\bm{w}})\| \leq M_{meta}\|{\bm{u}} - {\bm{w}}\|$, where $M_{meta} = (1+\alpha M)^{2K}M + C (b + \mathbb{E}_{i}\|\nabla \ell({\bm{w}}; \xi^q_i)\|)$, $b= \frac{1}{n} \sum_{i=1}^n b_i$ and $C =

Figures (9)

  • Figure 1: Average testing accuracies for training clients on mini-ImageNet.
  • Figure 2: Average testing accuracies for unseen clients on mini-ImageNet.
  • Figure 3: Average testing accuracy with communication cost for training clients on Meta-Datasets under 5-shot setting.
  • Figure 4: Average testing accuracy with communication cost for unseen clients on Meta-Datasets under 5-shot setting.
  • Figure 5: Example communication networks used in the experiments.
  • ...and 4 more figures

Theorems & Definitions (20)

  • Proposition 4.2
  • Theorem 4.3
  • Definition 4.4: Network Differential Privacy cyffers2022privacy
  • Theorem 4.5
  • Lemma E.1
  • proof
  • Lemma E.2
  • proof
  • Lemma E.3
  • proof : Proof for Lemma \ref{['lem:n1']}
  • ...and 10 more