Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Learning with Reduced Information Leakage and Computation

Tongxin Yin, Xuwei Tan, Xueru Zhang, Mohammad Mahdi Khalili, Mingyan Liu

TL;DR

This work introduces Upcycled-FL, a plug-in strategy for federated learning that reduces total information leakage and computation by applying a first-order approximation at every even update, so half of the iterations avoid local data usage. The method can be combined with existing FL algorithms (e.g., FedProx) and comes with convergence guarantees under standard FL assumptions. The authors extend Upcycled-FL to private FL by employing output and objective perturbations and show, via moments accountant and related analyses, that privacy losses accumulate only on odd iterations, allowing smaller perturbations and improved privacy-accuracy trade-offs. Extensive experiments on synthetic and real data (FEMNIST, Sent140) demonstrate robustness to heterogeneity and consistent privacy-accuracy gains without prohibitive runtime, illustrating practical impact for privacy-conscious distributed learning.

Abstract

Federated learning (FL) is a distributed learning paradigm that allows multiple decentralized clients to collaboratively learn a common model without sharing local data. Although local data is not exposed directly, privacy concerns nonetheless exist as clients' sensitive information can be inferred from intermediate computations. Moreover, such information leakage accumulates substantially over time as the same data is repeatedly used during the iterative learning process. As a result, it can be particularly difficult to balance the privacy-accuracy trade-off when designing privacy-preserving FL algorithms. This paper introduces Upcycled-FL, a simple yet effective strategy that applies first-order approximation at every even round of model update. Under this strategy, half of the FL updates incur no information leakage and require much less computational and transmission costs. We first conduct the theoretical analysis on the convergence (rate) of Upcycled-FL and then apply two perturbation mechanisms to preserve privacy. Extensive experiments on both synthetic and real-world data show that the Upcycled-FL strategy can be adapted to many existing FL frameworks and consistently improve the privacy-accuracy trade-off.

Federated Learning with Reduced Information Leakage and Computation

TL;DR

This work introduces Upcycled-FL, a plug-in strategy for federated learning that reduces total information leakage and computation by applying a first-order approximation at every even update, so half of the iterations avoid local data usage. The method can be combined with existing FL algorithms (e.g., FedProx) and comes with convergence guarantees under standard FL assumptions. The authors extend Upcycled-FL to private FL by employing output and objective perturbations and show, via moments accountant and related analyses, that privacy losses accumulate only on odd iterations, allowing smaller perturbations and improved privacy-accuracy trade-offs. Extensive experiments on synthetic and real data (FEMNIST, Sent140) demonstrate robustness to heterogeneity and consistent privacy-accuracy gains without prohibitive runtime, illustrating practical impact for privacy-conscious distributed learning.

Abstract

Federated learning (FL) is a distributed learning paradigm that allows multiple decentralized clients to collaboratively learn a common model without sharing local data. Although local data is not exposed directly, privacy concerns nonetheless exist as clients' sensitive information can be inferred from intermediate computations. Moreover, such information leakage accumulates substantially over time as the same data is repeatedly used during the iterative learning process. As a result, it can be particularly difficult to balance the privacy-accuracy trade-off when designing privacy-preserving FL algorithms. This paper introduces Upcycled-FL, a simple yet effective strategy that applies first-order approximation at every even round of model update. Under this strategy, half of the FL updates incur no information leakage and require much less computational and transmission costs. We first conduct the theoretical analysis on the convergence (rate) of Upcycled-FL and then apply two perturbation mechanisms to preserve privacy. Extensive experiments on both synthetic and real-world data show that the Upcycled-FL strategy can be adapted to many existing FL frameworks and consistently improve the privacy-accuracy trade-off.
Paper Structure (35 sections, 9 theorems, 58 equations, 9 figures, 5 tables, 2 algorithms)

This paper contains 35 sections, 9 theorems, 58 equations, 9 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Proposed Method: Upcycled-FL
  5. Main idea.
  6. Upcycling model update.
  7. Discussion.
  8. Convergence Analysis
  9. Private Upcycled-FL
  10. Experiments
  11. Datasets and Networks
  12. Experimental setup
  13. Results
  14. Convergence and Heterogeneity.
  15. Privacy-Accuracy Trade-off.
  16. ...and 20 more sections

Key Result

Lemma 5.3

$\forall i$, there exists $B$ such that $F_i$ is $B$-dissimilar if $||\nabla F_i(\omega) -\nabla f(\omega)||\leq \kappa_i, \forall \omega$ for some $\kappa_i$.

Figures (9)

  • Figure 1: Upcycled-FL can be considered from two perspectives: (i) it can be regarded as reusing the intermediate updates of local models to reduce the total information leakage; or (ii) it can be regarded as a global aggregation method with larger global update, which accelerates the learning process with the same information leakage under the same training iterations.
  • Figure 2: Comparison on average loss and standard deviation between Upcycled-FL methods and original FL algorithms in the non-private setting under the approximate same training time. The training time refers to the time needed for a given number of iterations. Upcycled-FL does not require an update in the even iterations, allowing Upcycled-FL to train with doubled iterations.
  • Figure 3: Comparison on average loss and standard deviation of private Upcycled-FL and private FL methods using output perturbation. The noise parameter $\sigma$ is 1.0 for all baselines, while $\sigma$ of the Upcycled version is set to 0.8. Taking the iid dataset as an example, $\Bar{\epsilon}=1.40$ for the Upcycled version, which ensures stronger privacy than the original method with $\Bar{\epsilon}=1.59$.
  • Figure 4: Comparison on average loss and standard deviation of private Upcycled-FL and private FL methods using objective perturbation. Under objective perturbation, noise parameter $\alpha$ is 10 for all baselines, while $\alpha$ of the Upcycled version is set to 20 to ensure stronger privacy than the original versions. Taking the iid dataset as an example, $\Bar{\epsilon}$ associated with these noise parameters is 7.36 for FedProx and 7.25 for Upcycled-FedProx (when $\mu$ = 0.5).
  • Figure 5: Convergence of Upcycled-FL and regular FL methods with the approximate same training time under 90% Straggler.
  • ...and 4 more figures

Theorems & Definitions (16)

  • Example 4.1: FedAvg mcmahan2017communication under Upcycled-FL strategy
  • Example 4.2: FedProx li2020federated under Upcycled-FL strategy
  • Definition 5.1: $B$-Dissimilarity li2020federated
  • Lemma 5.3
  • Lemma 5.5
  • Theorem 5.6: Convergence rate of Upcycled-FedProx
  • Corollary 5.8: Convergence to the stationary point
  • Lemma 6.1
  • Theorem 6.2
  • Theorem 6.3
  • ...and 6 more