Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Methods for Unknown Piecewise Smooth Problems I: Convex Optimization

Zhenwei Lin, Zhe Zhang

TL;DR

This work introduces an optimal and nearly parameter-free algorithm for minimizing piecewise smooth (PWS) convex functions under the quadratic growth condition, where the locations and structure of the smooth regions are entirely unknown.

Abstract

We introduce an optimal and nearly parameter-free algorithm for minimizing piecewise smooth (PWS) convex functions under the quadratic growth (QG) condition, where the locations and structure of the smooth regions are entirely \textit{unknown}. Our algorithm, \apex{} (Accelerated Prox-Level method for Exploring Piecewise Smoothness), is an accelerated bundle-level method designed to adaptively exploit the underlying PWS structure. APEX enjoys optimal theoretical guarantees, achieving a tight oracle complexity bound that matches the lower bound established in this work for convex PWS optimization. Furthermore, APEX generates a verifiable and accurate termination certificate, enabling a robust, almost parameter-free implementation. To the best of our knowledge, APEX is the first algorithm to simultaneously achieve the optimal convergence rate for PWS optimization and provide certificate guarantees.

Optimal Methods for Unknown Piecewise Smooth Problems I: Convex Optimization

TL;DR

This work introduces an optimal and nearly parameter-free algorithm for minimizing piecewise smooth (PWS) convex functions under the quadratic growth condition, where the locations and structure of the smooth regions are entirely unknown.

Abstract

We introduce an optimal and nearly parameter-free algorithm for minimizing piecewise smooth (PWS) convex functions under the quadratic growth (QG) condition, where the locations and structure of the smooth regions are entirely \textit{unknown}. Our algorithm, \apex{} (Accelerated Prox-Level method for Exploring Piecewise Smoothness), is an accelerated bundle-level method designed to adaptively exploit the underlying PWS structure. APEX enjoys optimal theoretical guarantees, achieving a tight oracle complexity bound that matches the lower bound established in this work for convex PWS optimization. Furthermore, APEX generates a verifiable and accurate termination certificate, enabling a robust, almost parameter-free implementation. To the best of our knowledge, APEX is the first algorithm to simultaneously achieve the optimal convergence rate for PWS optimization and provide certificate guarantees.
Paper Structure (23 sections, 17 theorems, 99 equations, 6 figures, 5 tables, 6 algorithms)

This paper contains 23 sections, 17 theorems, 99 equations, 6 figures, 5 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Absence of optimal global convergence rates for unknown PWS optimization.
  3. Sensitivity to unavailable problem-specific parameters.
  4. Lack of efficient, accurate, and verifiable certificate generation.
  5. Related Work
  6. Algorithms for nonsmooth optimization:
  7. Certificate guarantees:
  8. Outline
  9. Notations
  10. Preliminaries
  11. Accelerated Prox-level method for piecewise smooth optimization
  12. Normalized Wolfe certificate
  13. Restarted $\text{APEX}$ for known $\mu^{*}$
  14. Restarted $\text{APEX}$ for unknown $\mu^{*}$
  15. Lower oracle complexity bound for $\mu$-QG and $\left(k,L\right)$-$\text{PWS}$ optimization
  16. ...and 8 more sections

Key Result

Proposition 3.1

Suppose function $f$ is convex, $\left(k,L\right)$-$\text{PWS}$ and the sequence $\left\{ \hat{x}^{t}\right\}$ generated by Algorithm alg:apex satisfies $f\left(\hat{x}^{t}\right)\geq\tilde{l}$ for all $t\geq1$ and that every subproblem eq:subproblems of Algorithm alg:apex is feasible. Let $x^{t}:=

Figures (6)

  • Figure 1: Convergence comparison between BL, APL, and Restarted APL on a randomly generated $\texttt{MAXQUAD}$ problem.
  • Figure 2: Worst and average empirical Lipschitz smoothness constant comparison when running Polyak Step Size in Example \ref{['ex:intro-core']} with parameter $i=5$, $M=10^{10}$.
  • Figure 3: $\max\left\{ {x}^2, x\right\}$ is (2, 2)-$\text{PWS}$ and region $X_1$ is disconnected.
  • Figure 4: Trajectory comparison of Polyak step size and BL method on minimizing $\left\Vert x\right\Vert ^2+\vert x_{(1)}\vert$ starting from $x^1=(0.01, 0.15)$.
  • Figure 5: Absolute error of upper bound and lower bound of rAPEX v.s. Iteration $\texttt{MAXQUAD}$ with $k=50,L=10$ in Table \ref{['tab:com_rapex_max_quad']}.
  • ...and 1 more figures

Theorems & Definitions (41)

  • Definition 2.1: $(k,L)$-Piecewise smoothness, $\left(k,L\right)$-PWS
  • Definition 2.2: $\mu^*$-Quadratic growth, $\mu^*$-QG
  • Proposition 3.1
  • proof
  • Theorem 3.1
  • proof
  • Proposition 3.2
  • proof
  • Definition 4.1: Normalized Wolfe certificate, $\text{W-certificate}$
  • Lemma 4.1
  • ...and 31 more