Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning

Shayan Mohajer Hamidi, Ali Bereyhi, Saba Asaad, H. Vincent Poor

TL;DR

GP-FL tackles the challenge of fast, second-order federated learning over wireless channels by permitting the PS to estimate the global Hessian from noisy gradient sums using a Gaussian-process prior. The approach replaces deterministic Hessian computation with a model-based, posterior sampling estimate that leverages a window of recent aggregations, enabling a descent direction $\tilde{\mathfrak{d}}_t = -\hat{\boldsymbol{B}}_t^{-1} \tilde{\boldsymbol{g}}_t$ in each round. It proves a linear-quadratic convergence rate and demonstrates superior performance across LIBSVM, Fashion-MNIST, and CIFAR datasets under non-iid and noisy uplink conditions. The method significantly reduces communication overhead while maintaining convergence speed, highlighting the practical impact of probabilistic Hessian estimation in wireless FL.

Abstract

Second-order methods are widely adopted to improve the convergence rate of learning algorithms. In federated learning (FL), these methods require the clients to share their local Hessian matrices with the parameter server (PS), which comes at a prohibitive communication cost. A classical solution to this issue is to approximate the global Hessian matrix from the first-order information. Unlike in idealized networks, this solution does not perform effectively in over-the-air FL settings, where the PS receives noisy versions of the local gradients. This paper introduces a novel second-order FL framework tailored for wireless channels. The pivotal innovation lies in the PS's capability to directly estimate the global Hessian matrix from the received noisy local gradients via a non-parametric method: the PS models the unknown Hessian matrix as a Gaussian process, and then uses the temporal relation between the gradients and Hessian along with the channel model to find a stochastic estimator for the global Hessian matrix. We refer to this method as Gaussian process-based Hessian modeling for wireless FL (GP-FL) and show that it exhibits a linear-quadratic convergence rate. Numerical experiments on various datasets demonstrate that GP-FL outperforms all classical baseline first and second order FL approaches.

GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning

TL;DR

GP-FL tackles the challenge of fast, second-order federated learning over wireless channels by permitting the PS to estimate the global Hessian from noisy gradient sums using a Gaussian-process prior. The approach replaces deterministic Hessian computation with a model-based, posterior sampling estimate that leverages a window of recent aggregations, enabling a descent direction in each round. It proves a linear-quadratic convergence rate and demonstrates superior performance across LIBSVM, Fashion-MNIST, and CIFAR datasets under non-iid and noisy uplink conditions. The method significantly reduces communication overhead while maintaining convergence speed, highlighting the practical impact of probabilistic Hessian estimation in wireless FL.

Abstract

Second-order methods are widely adopted to improve the convergence rate of learning algorithms. In federated learning (FL), these methods require the clients to share their local Hessian matrices with the parameter server (PS), which comes at a prohibitive communication cost. A classical solution to this issue is to approximate the global Hessian matrix from the first-order information. Unlike in idealized networks, this solution does not perform effectively in over-the-air FL settings, where the PS receives noisy versions of the local gradients. This paper introduces a novel second-order FL framework tailored for wireless channels. The pivotal innovation lies in the PS's capability to directly estimate the global Hessian matrix from the received noisy local gradients via a non-parametric method: the PS models the unknown Hessian matrix as a Gaussian process, and then uses the temporal relation between the gradients and Hessian along with the channel model to find a stochastic estimator for the global Hessian matrix. We refer to this method as Gaussian process-based Hessian modeling for wireless FL (GP-FL) and show that it exhibits a linear-quadratic convergence rate. Numerical experiments on various datasets demonstrate that GP-FL outperforms all classical baseline first and second order FL approaches.

Paper Structure

This paper contains 37 sections, 5 theorems, 75 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Communication-Efficient FL
  3. Second-Order Wireless FL
  4. Motivation and Contributions
  5. Related Works
  6. Notation
  7. Organization
  8. Preliminaries
  9. First-Order Federated Learning
  10. Newton's Method in Federated Learning
  11. Quasi-Newton Search Directions
  12. GP-FL: Quasi-Newton Method in Wireless Networks
  13. GP-FL: Over-the-Air Aggregation
  14. Uplink Channel Model
  15. Model Aggregation via AirComp
  16. ...and 22 more sections

Key Result

Lemma 1

Consider the Gaussian model for $\bold{z}_t$ given in eq:zt. The entry $\Tilde{b}_t$ conditional to observation $\bold{o}_t$ is distributed Gaussian with mean $\zeta (\bold{o}_t)$ and variance $\psi (\bold{o}_t)$ that are given by where $\bold{a}_t$ is the solution to $\bold{K}_t \bold{a}_t = \boldsymbol{\phi}_t$ which can be computed via the conjugate gradient algorithm.

Figures (1)

  • Figure 1: Test accuracy vs. communication rounds for GP-FL and baseline methods across different datasets.

Theorems & Definitions (17)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Definition 1: Radial basis kernel
  • Lemma 1: Posterior of quasi-Newton matrix
  • proof
  • Remark 5
  • Remark 6
  • Definition 2: $\delta$-approximate
  • ...and 7 more