Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HBNET-GIANT: A communication-efficient accelerated Newton-type fully distributed optimization algorithm

Souvik Das, Luca Schenato, Subhrakanti Dey

TL;DR

This work addresses fully distributed optimization over a network of agents with $L$-smooth and $\mu$-strongly convex local objectives by introducing HbNet-GIANT, a Newton-type method augmented with heavy-ball momentum. The method maintains communication efficiency at $O(n p)$ per iteration through gradient-tracking and a second-order oracle that exchanges $S$ and related trackers rather than full Hessians. The authors prove global linear convergence under verifiable conditions on the step-size $\eta$ and momentum $\beta$, with convergence characterized by the spectral radius $\rho(A(\eta,\beta))$, and demonstrate accelerated convergence numerically compared to state-of-the-art baselines. This work lays the groundwork for a broader class of second-order Newton-type algorithms with momentum in fully distributed settings and motivates further theoretical exploration of local acceleration in distributed optimization.

Abstract

This article presents a second-order fully distributed optimization algorithm, HBNET-GIANT, driven by heavy-ball momentum, for $L$-smooth and $μ$-strongly convex objective functions. A rigorous convergence analysis is performed, and we demonstrate global linear convergence under certain sufficient conditions. Through extensive numerical experiments, we show that HBNET-GIANT with heavy-ball momentum achieves acceleration, and the corresponding rate of convergence is strictly faster than its non-accelerated version, NETWORK-GIANT. Moreover, we compare HBNET-GIANT with several state-of-the-art algorithms, both momentum-based and without momentum, and report significant performance improvement in convergence to the optimum. We believe that this work lays the groundwork for a broader class of second-order Newton-type algorithms with momentum and motivates further investigation into open problems, including an analytical proof of local acceleration in the fully distributed setting for convex optimization problems.

HBNET-GIANT: A communication-efficient accelerated Newton-type fully distributed optimization algorithm

TL;DR

This work addresses fully distributed optimization over a network of agents with -smooth and -strongly convex local objectives by introducing HbNet-GIANT, a Newton-type method augmented with heavy-ball momentum. The method maintains communication efficiency at per iteration through gradient-tracking and a second-order oracle that exchanges and related trackers rather than full Hessians. The authors prove global linear convergence under verifiable conditions on the step-size and momentum , with convergence characterized by the spectral radius , and demonstrate accelerated convergence numerically compared to state-of-the-art baselines. This work lays the groundwork for a broader class of second-order Newton-type algorithms with momentum in fully distributed settings and motivates further theoretical exploration of local acceleration in distributed optimization.

Abstract

This article presents a second-order fully distributed optimization algorithm, HBNET-GIANT, driven by heavy-ball momentum, for -smooth and -strongly convex objective functions. A rigorous convergence analysis is performed, and we demonstrate global linear convergence under certain sufficient conditions. Through extensive numerical experiments, we show that HBNET-GIANT with heavy-ball momentum achieves acceleration, and the corresponding rate of convergence is strictly faster than its non-accelerated version, NETWORK-GIANT. Moreover, we compare HBNET-GIANT with several state-of-the-art algorithms, both momentum-based and without momentum, and report significant performance improvement in convergence to the optimum. We believe that this work lays the groundwork for a broader class of second-order Newton-type algorithms with momentum and motivates further investigation into open problems, including an analytical proof of local acceleration in the fully distributed setting for convex optimization problems.

Paper Structure

This paper contains 11 sections, 3 theorems, 38 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Problem description
  4. Standing assumptions
  5. $\textsc{HbNet-GIANT}$ and its analysis
  6. Key steps of $\textsc{HbNet-GIANT}$
  7. Features of $\textsc{HbNet-GIANT}$
  8. Convergence of $\textsc{HbNet-GIANT}$
  9. Experiments
  10. Conclusion
  11. Proof of Lemma \ref{['lem:portmanteuau 1']}

Key Result

Lemma III.1

Consider Algorithm alg:sec_ord_comp along with its associated notations. Suppose that Assumptions assum:on graph and assum:Standard assumptions hold. Then the following assertions hold:

Figures (2)

  • Figure 1: Comparison of $\textsc{HbNet-GIANT}$ with GradTrack, $\mathcal{A}\mathcal{B}m$, Acc-DNGD-SC, and Network-GIANT for $\widetilde{d}$-regular expander graph, with $\widetilde{d} = 14$.
  • Figure 2: Comparison of $\textsc{HbNet-GIANT}$ with GradTrack, $\mathcal{A}\mathcal{B}m$, Acc-DNGD-SC, and Network-GIANT for Erdős-Rényi graph, with $\widetilde{p} = 0.3$.

Theorems & Definitions (8)

  • Lemma III.1
  • Proposition III.2
  • proof
  • Theorem III.3
  • proof
  • Remark III.1
  • Remark III.2
  • Remark III.3