Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DP-FEDSOFIM: Differentially Private Federated Stochastic Optimization using Regularized Fisher Information Matrix

Sidhant R. Nair, Tanmay Sen, Mrinmay Sen

TL;DR

This work proposes DP-FedSOFIM, a server-side second-order optimization framework that leverages the Fisher Information Matrix (FIM) as a natural gradient preconditioner while requiring only O(d) memory per client and proves that the server-side preconditioning preserves (epsilon, delta)-differential privacy through the post-processing theorem.

Abstract

Differentially private federated learning (DP-FL) suffers from slow convergence under tight privacy budgets due to the overwhelming noise introduced to preserve privacy. While adaptive optimizers can accelerate convergence, existing second-order methods such as DP-FedNew require O(d^2) memory at each client to maintain local feature covariance matrices, making them impractical for high-dimensional models. We propose DP-FedSOFIM, a server-side second-order optimization framework that leverages the Fisher Information Matrix (FIM) as a natural gradient preconditioner while requiring only O(d) memory per client. By employing the Sherman-Morrison formula for efficient matrix inversion, DP-FedSOFIM achieves O(d) computational complexity per round while maintaining the convergence benefits of second-order methods. Our analysis proves that the server-side preconditioning preserves (epsilon, delta)-differential privacy through the post-processing theorem. Empirical evaluation on CIFAR-10 demonstrates that DP-FedSOFIM achieves superior test accuracy compared to first-order baselines across multiple privacy regimes.

DP-FEDSOFIM: Differentially Private Federated Stochastic Optimization using Regularized Fisher Information Matrix

TL;DR

This work proposes DP-FedSOFIM, a server-side second-order optimization framework that leverages the Fisher Information Matrix (FIM) as a natural gradient preconditioner while requiring only O(d) memory per client and proves that the server-side preconditioning preserves (epsilon, delta)-differential privacy through the post-processing theorem.

Abstract

Differentially private federated learning (DP-FL) suffers from slow convergence under tight privacy budgets due to the overwhelming noise introduced to preserve privacy. While adaptive optimizers can accelerate convergence, existing second-order methods such as DP-FedNew require O(d^2) memory at each client to maintain local feature covariance matrices, making them impractical for high-dimensional models. We propose DP-FedSOFIM, a server-side second-order optimization framework that leverages the Fisher Information Matrix (FIM) as a natural gradient preconditioner while requiring only O(d) memory per client. By employing the Sherman-Morrison formula for efficient matrix inversion, DP-FedSOFIM achieves O(d) computational complexity per round while maintaining the convergence benefits of second-order methods. Our analysis proves that the server-side preconditioning preserves (epsilon, delta)-differential privacy through the post-processing theorem. Empirical evaluation on CIFAR-10 demonstrates that DP-FedSOFIM achieves superior test accuracy compared to first-order baselines across multiple privacy regimes.
Paper Structure (38 sections, 3 theorems, 62 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 38 sections, 3 theorems, 62 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Differentially Private Federated Learning
  4. Adaptive and Second-Order Optimizers in Federated Learning
  5. Efficient Matrix Inversion Techniques
  6. Methodology
  7. Problem Setup and Notation
  8. Baseline: DP-FedGD with Gradient Clipping and Noise Addition
  9. DP-FedSOFIM: Server-Side Second-Order Preconditioning
  10. Fisher Information Matrix as Natural Gradient Preconditioner
  11. Server-Side FIM Construction from Noisy Gradients
  12. Efficient Inversion via Sherman-Morrison Formula
  13. Complete DP-FedSOFIM Algorithm
  14. Convergence and Privacy Analysis
  15. Convergence Guarantees
  16. ...and 23 more sections

Key Result

Lemma 4.5

The inverse Fisher Information Matrix $H^t = (\hat{\mathcal{F}}^t)^{-1}$ computed via the Sherman-Morrison formula satisfies: where $\preceq$ denotes the Loewner order (positive semidefinite ordering). This ensures the condition number $\kappa(H^t) \leq \frac{\rho + \|M^t\|_2^2}{\rho}$.

Figures (1)

  • Figure 1: Convergence comparison of DP-FedSOFIM (solid) versus DP-FedGD (dashed) across privacy regimes on CIFAR-10. DP-FedSOFIM achieves consistent final accuracy gains (+0.42% to +3.12%), with early deficits under tight privacy ($\epsilon \leq 2$) overcome by round 30, and immediate dominance under relaxed privacy ($\epsilon \geq 5$).

Theorems & Definitions (12)

  • Definition 3.1: ($\epsilon, \delta$)-Differential Privacy
  • Remark 3.2: Record-Level Adjacency
  • Remark 4.3: Justification for Frozen Features
  • Lemma 4.5: Eigenvalue Bounds of SOFIM Preconditioner
  • proof
  • Remark 4.6
  • Theorem 4.7: Convergence of DP-FedSOFIM
  • proof
  • Remark 4.8
  • Lemma 4.9: Privacy Preservation via Post-Processing
  • ...and 2 more