Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropic Mirror Descent for Linear Systems: Polyak's Stepsize and Implicit Bias

Yura Malitsky, Alexander Posch

TL;DR

This work analyzes entropic mirror descent for solving linear systems, addressing the challenge of unbounded domains with an adaptive Polyak-type stepsize. It establishes convergence for nonnegative and general linear systems, derives sublinear rates and, under a strict positive separation of the solution, linear rates, and demonstrates linear convergence for the EG$\pm$ variant. The paper further extends the framework to Hadamard Descent+ and to arbitrary convex $L$-smooth functions with known optima, broadening applicability beyond linear systems. By connecting MD with multiplicative updates and implicit bias toward $\ell_1$-favoring solutions, the results provide provable, adaptive convergence guarantees for a class of exponentiated-gradient methods and offer practical tools for sparse recovery and related problems.

Abstract

This paper focuses on applying entropic mirror descent to solve linear systems, where the main challenge for the convergence analysis stems from the unboundedness of the domain. To overcome this without imposing restrictive assumptions, we introduce a variant of Polyak-type stepsizes. Along the way, we strengthen the bound for $\ell_1$-norm implicit bias, obtain sublinear and linear convergence results, and generalize the convergence result to arbitrary convex $L$-smooth functions. We also propose an alternative method that avoids exponentiation, resembling the original Hadamard descent, but with provable convergence.

Entropic Mirror Descent for Linear Systems: Polyak's Stepsize and Implicit Bias

TL;DR

This work analyzes entropic mirror descent for solving linear systems, addressing the challenge of unbounded domains with an adaptive Polyak-type stepsize. It establishes convergence for nonnegative and general linear systems, derives sublinear rates and, under a strict positive separation of the solution, linear rates, and demonstrates linear convergence for the EG variant. The paper further extends the framework to Hadamard Descent+ and to arbitrary convex -smooth functions with known optima, broadening applicability beyond linear systems. By connecting MD with multiplicative updates and implicit bias toward -favoring solutions, the results provide provable, adaptive convergence guarantees for a class of exponentiated-gradient methods and offer practical tools for sparse recovery and related problems.

Abstract

This paper focuses on applying entropic mirror descent to solve linear systems, where the main challenge for the convergence analysis stems from the unboundedness of the domain. To overcome this without imposing restrictive assumptions, we introduce a variant of Polyak-type stepsizes. Along the way, we strengthen the bound for -norm implicit bias, obtain sublinear and linear convergence results, and generalize the convergence result to arbitrary convex -smooth functions. We also propose an alternative method that avoids exponentiation, resembling the original Hadamard descent, but with provable convergence.
Paper Structure (24 sections, 13 theorems, 95 equations, 2 figures)

This paper contains 24 sections, 13 theorems, 95 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Implicit bias.
  4. Hadamard parametrization.
  5. Mirror descent.
  6. Polyak's stepsize.
  7. Roadmap.
  8. Notation.
  9. Nonnegative Linear System
  10. Implicit Bias of Entropic Mirror Descent
  11. Convergence
  12. Linear System
  13. Linear Convergence Rates
  14. EG± Converges Linearly
  15. Generalization
  16. ...and 9 more sections

Key Result

Proposition 1

For any $A \neq 0 \in \mathbb{R}^{m \times n}$ and a stepsize $\alpha > 0$, there exists a vector $\boldsymbol{b} \in \mathbb{R}^m$ such that, for mirror descent eq:mdmd applied to $f(\boldsymbol{x}) = \frac{1}{2} \|A\boldsymbol{x} - \boldsymbol{b} \|^2$, any solution is an unstable fixed point.

Figures (2)

  • Figure 1: Left: Cumulative minimum function value $f(\bar{\boldsymbol{x}}_k)-f^*$ for MD-const., MD-Polyak, HD-Polyak, and HD+Polyak. Right: Bregman divergence to the estimated limit.
  • Figure 2: Cumulative minimum function value $f(\bar{\boldsymbol{x}}_k)-f^*$ for mirror descent with Polyak’s stepsize under various initialization. Left: Sparse solution ($\|\boldsymbol{z}_1\|_0=30$). Right: Dense solution.

Theorems & Definitions (31)

  • Proposition 1: Instability of Mirror Descent with Constant Stepsizes independent of $\boldsymbol{b}$
  • Proposition 2
  • proof
  • Remark 3
  • Example 4
  • Theorem 5
  • proof
  • Remark 6
  • Theorem 7
  • Lemma 8: Folklore
  • ...and 21 more