Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linear Convergence and Error Bounds for Optimization Without Strong Convexity

Kira van Treek, Javier F. Peña, Juan C. Vera, Luis F. Zuluaga

TL;DR

The paper addresses linear convergence of fixed-point iterations used to solve optimization problems without relying on strong convexity. It proves that linear convergence of averaged FPIs is equivalent to a local error-bound condition and shows that piecewise linear operators satisfy this bound, enabling practical linear rates for algorithms like ADMM and Douglas–Rachford. The results yield data-independent rates for linear optimization and rate bounds for quadratic optimization that depend only on the problem's condition number, via Hoffman-type constants. This framework provides a robust, general tool for analyzing convergence across LO, QO, and PLQ problems, with direct implications for common optimization algorithms and related fixed-point formulations.

Abstract

Many optimization algorithms$\unicode{x2013}$including gradient descent, proximal methods, and operator splitting techniques$\unicode{x2013}$can be formulated as fixed-point iterations (FPI) of continuous operators. When these operators are averaged, convergence to a fixed point is guaranteed when one exists, but the convergence is generally sublinear. Recent results establish linear convergence of FPI for averaged operators under certain conditions. However, such conditions do not apply to common classes of operators, such as those arising in piecewise linear and quadratic optimization problems. In this work, we prove that a local error-bound condition is both necessary and sufficient for the linear convergence of FPI applied to averaged operators. We provide explicit bounds on the convergence rate and show how these relate to the constants in the error-bound condition. Our main result demonstrates that piecewise linear operators satisfy local error bounds, ensuring linear convergence of the associated optimization algorithms. This leads to a general and practical framework for analyzing convergence behavior in algorithms such as ADMM and Douglas-Rachford in the absence of strong convexity. In particular, we obtain convergence rates that are independent of problem data for linear optimization, and depend only on the condition number of the objective for quadratic optimization.

Linear Convergence and Error Bounds for Optimization Without Strong Convexity

TL;DR

The paper addresses linear convergence of fixed-point iterations used to solve optimization problems without relying on strong convexity. It proves that linear convergence of averaged FPIs is equivalent to a local error-bound condition and shows that piecewise linear operators satisfy this bound, enabling practical linear rates for algorithms like ADMM and Douglas–Rachford. The results yield data-independent rates for linear optimization and rate bounds for quadratic optimization that depend only on the problem's condition number, via Hoffman-type constants. This framework provides a robust, general tool for analyzing convergence across LO, QO, and PLQ problems, with direct implications for common optimization algorithms and related fixed-point formulations.

Abstract

Many optimization algorithmsincluding gradient descent, proximal methods, and operator splitting techniquescan be formulated as fixed-point iterations (FPI) of continuous operators. When these operators are averaged, convergence to a fixed point is guaranteed when one exists, but the convergence is generally sublinear. Recent results establish linear convergence of FPI for averaged operators under certain conditions. However, such conditions do not apply to common classes of operators, such as those arising in piecewise linear and quadratic optimization problems. In this work, we prove that a local error-bound condition is both necessary and sufficient for the linear convergence of FPI applied to averaged operators. We provide explicit bounds on the convergence rate and show how these relate to the constants in the error-bound condition. Our main result demonstrates that piecewise linear operators satisfy local error bounds, ensuring linear convergence of the associated optimization algorithms. This leads to a general and practical framework for analyzing convergence behavior in algorithms such as ADMM and Douglas-Rachford in the absence of strong convexity. In particular, we obtain convergence rates that are independent of problem data for linear optimization, and depend only on the condition number of the objective for quadratic optimization.

Paper Structure

This paper contains 13 sections, 12 theorems, 111 equations, 1 table, 2 algorithms.

Table of Contents

  1. Introduction.
  2. Linear convergence and error bound equivalence.
  3. Piecewise linear fixed-point operators.
  4. Piecewise linear-quadratic optimization.
  5. Preliminaries.
  6. Douglas-Rachford Algorithm.
  7. Peaceman-Rachford Algorithm.
  8. ADMM.
  9. From fixed-points to optimal points
  10. FPI algorithms for PLQ optimization.
  11. ADMM/DR for Linear Optimization.
  12. ADMM/DR for Quadratic Optimization.
  13. Final Remarks.

Key Result

Theorem 1

Let $0 < \alpha < 1$ and $F:\operatorname{\mathbb R}^n\to\operatorname{\mathbb R}^n$ be an $\alpha$-averaged operator, such that $F$ has fixed-points. Given $x_0\in\operatorname{\mathbb R}^n$, let $x_{k+1}=F(x_k)$ for $k=0,1,\dots$. Then

Theorems & Definitions (20)

  • Example 1: Gradient descent for unconstrained convex optimization
  • Example 2: Gradient descent is an averaged operator
  • Theorem 1: Convergence of averaged operators
  • Lemma 1
  • Theorem 2: Error bound implies linear convergence
  • Proposition 3: Linear convergence implies error bound
  • Corollary 1: Error-bound condition and linear convergence bounds
  • Example 3: Convergence rates and error bounds for Gradient descent
  • Example 4: Convergence rates and error bounds for a rotation
  • Definition 1: Compatible collection
  • ...and 10 more