Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information Geometry of Exponentiated Gradient: Convergence beyond L-Smoothness

Yara Elshiaty, Ferdinand Vanmaele, Stefania Petra

TL;DR

The paper addresses minimizing smooth, potentially nonconvex functions on the positive orthant, motivated by Poisson inverse problems. It reframes exponentiated gradient (EG) as Riemannian gradient descent on the Poisson geometry using the $e$-Exp retraction, enabling a Riemannian Armijo line search. The authors prove global convergence under weak conditions and establish finite termination of the line search, highlighting the KL-divergence’s self-concordant-like role in the analysis. Numerical experiments in tomographic reconstruction show that EG with Armijo line search converges faster than interior-point geometry-based RGD, and an accelerated EG variant via geometric CG demonstrates practical benefits.

Abstract

We study the minimization of smooth, possibly nonconvex functions over the positive orthant, a key setting in Poisson inverse problems, using the exponentiated gradient (EG) method. Interpreting EG as Riemannian gradient descent (RGD) with the $e$-Exp map from information geometry as a retraction, we prove global convergence under weak assumptions -- without the need for $L$-smoothness -- and finite termination of Riemannian Armijo line search. Numerical experiments, including an accelerated variant, highlight EG's practical advantages, such as faster convergence compared to RGD based on interior-point geometry.

Information Geometry of Exponentiated Gradient: Convergence beyond L-Smoothness

TL;DR

The paper addresses minimizing smooth, potentially nonconvex functions on the positive orthant, motivated by Poisson inverse problems. It reframes exponentiated gradient (EG) as Riemannian gradient descent on the Poisson geometry using the -Exp retraction, enabling a Riemannian Armijo line search. The authors prove global convergence under weak conditions and establish finite termination of the line search, highlighting the KL-divergence’s self-concordant-like role in the analysis. Numerical experiments in tomographic reconstruction show that EG with Armijo line search converges faster than interior-point geometry-based RGD, and an accelerated EG variant via geometric CG demonstrates practical benefits.

Abstract

We study the minimization of smooth, possibly nonconvex functions over the positive orthant, a key setting in Poisson inverse problems, using the exponentiated gradient (EG) method. Interpreting EG as Riemannian gradient descent (RGD) with the -Exp map from information geometry as a retraction, we prove global convergence under weak assumptions -- without the need for -smoothness -- and finite termination of Riemannian Armijo line search. Numerical experiments, including an accelerated variant, highlight EG's practical advantages, such as faster convergence compared to RGD based on interior-point geometry.

Paper Structure

This paper contains 5 sections, 8 theorems, 50 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Geometry of EG on the Positive Orthant
  3. EG with Riemannian Armijo line search
  4. Experiments
  5. Conclusion

Key Result

Proposition 2.1

Let $\mathcal{M} = \mathbb{R}^n_{++}$ be endowed with the Poisson Fisher-Rao metric eq:Poisson-geometry. Then, the eq:EG iteration is equivalent to

Figures (2)

  • Figure 4.1: Reconstructions across iterations. Comparison of EG with Armijo backtracking (top row), $g$-RGD with Armijo backtracking (second row), and IP $e$-RGD with a constant step size (third row) based on interior-point geometry, along with PR-type CG-accelerated EG for iterations $k = 10, 20, 50, 100, 300$. Since accelerated EG terminates at 148 iterations, its result at iteration 148 is shown for iteration 300.
  • Figure 4.2: Comparison of Algorithms.Left: Relative function values of EG (Poi $e$-RGD), its accelerated variant (Poi $e$-CG), and $g$-RGD and $e$-RGD with interior-point geometry. Accelerated EG outperforms and terminates after 148 iterations. Right: Average matrix-value operations across iterations. While IP $e$-RGD is cheaper without line search, EG still achieves better performance.

Theorems & Definitions (15)

  • Proposition 2.1: EG as RGD Raus2024
  • Remark 1
  • Theorem 3.1: Termination of \ref{['eq:R-Armijo']}
  • Corollary 3.2: Convergence of EG with Riemannian Armijo line search
  • Lemma 3.3
  • proof
  • Lemma 3.4: KL scaling property
  • proof
  • Corollary 3.5: Pinsker's type inequality for positive orthant
  • proof
  • ...and 5 more