Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Distance-adaptive Methods for Hölder Smooth and Convex Optimization

Yijin Ren, Haifeng Xu, Qi Deng

TL;DR

The paper tackles convex optimization where the objective has Hölder smoothness by introducing a parameter-free accelerated distance-adaptive method (AGDA) that estimates distance to the minimizer through distance-adaptive quantities and line-search guided updates, achieving anytime convergence without requiring $L_\nu$, $\nu$, or $D_0$. It further extends to a line-search-free stochastic variant (LF-AGDA) under a bounded-domain assumption, with convergence guarantees that mix local Hölder smoothness and stochastic noise. The proposed methods outperform existing universal or parameter-tuned schemes in deterministic and stochastic experiments, and exhibit robustness to mis-specified problem parameters, including the initial distance estimate $\bar{r}$. Overall, the work advances practical, adaptive first-order optimization for Hölder-smooth convex problems and provides empirically competitive results in nonconvex neural-network settings.

Abstract

This paper introduces new parameter-free first-order methods for convex optimization problems in which the objective function exhibits Hölder smoothness. Inspired by the recently proposed distance-over-gradient (DOG) technique, we propose an accelerated distance-adaptive method which achieves optimal anytime convergence rates for Hölder smooth problems without requiring prior knowledge of smoothness parameters or explicit parameter tuning. Importantly, our parameter-free approach removes the necessity of specifying target accuracy in advance, addressing a limitation found in the universal fast gradient methods (Nesterov, Yu. \textit{Mathematical Programming}, 2015). For convex stochastic optimization, we further present a parameter-free accelerated method that eliminates the need for line-search procedures. Preliminary experimental results highlight the effectiveness of our approach on convex nonsmooth problems and its advantages over existing parameter-free or accelerated methods.

Accelerated Distance-adaptive Methods for Hölder Smooth and Convex Optimization

TL;DR

The paper tackles convex optimization where the objective has Hölder smoothness by introducing a parameter-free accelerated distance-adaptive method (AGDA) that estimates distance to the minimizer through distance-adaptive quantities and line-search guided updates, achieving anytime convergence without requiring , , or . It further extends to a line-search-free stochastic variant (LF-AGDA) under a bounded-domain assumption, with convergence guarantees that mix local Hölder smoothness and stochastic noise. The proposed methods outperform existing universal or parameter-tuned schemes in deterministic and stochastic experiments, and exhibit robustness to mis-specified problem parameters, including the initial distance estimate . Overall, the work advances practical, adaptive first-order optimization for Hölder-smooth convex problems and provides empirically competitive results in nonconvex neural-network settings.

Abstract

This paper introduces new parameter-free first-order methods for convex optimization problems in which the objective function exhibits Hölder smoothness. Inspired by the recently proposed distance-over-gradient (DOG) technique, we propose an accelerated distance-adaptive method which achieves optimal anytime convergence rates for Hölder smooth problems without requiring prior knowledge of smoothness parameters or explicit parameter tuning. Importantly, our parameter-free approach removes the necessity of specifying target accuracy in advance, addressing a limitation found in the universal fast gradient methods (Nesterov, Yu. \textit{Mathematical Programming}, 2015). For convex stochastic optimization, we further present a parameter-free accelerated method that eliminates the need for line-search procedures. Preliminary experimental results highlight the effectiveness of our approach on convex nonsmooth problems and its advantages over existing parameter-free or accelerated methods.
Paper Structure (44 sections, 18 theorems, 163 equations, 5 figures, 5 algorithms)

This paper contains 44 sections, 18 theorems, 163 equations, 5 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. The accelerated distance-adaptive method
  5. Motivation
  6. Line search
  7. Stochastic optimization
  8. Experiments
  9. Deterministic setting
  10. Softmax
  11. Matrix game
  12. Stochastic setting
  13. Least-squares
  14. Robustness
  15. Non-convex neural network
  16. ...and 29 more sections

Key Result

Proposition 1

Suppose $f(\cdot)$ is locally Hölder smooth assumption 1:local smooth in $\mathcal{B}_{3D_0}(x^*)$. In Algorithm alg:agda, for any $k\geq0$, at least one of the following two conditions holds: Consequently, we have $\beta_{k+1}\leq \mathcal{O}(k^{\frac{3-3\nu}{2}})$. Moreover, the total number of iterations required by the line search in Algorithm alg:agda is $\sum_{k=0}^{K-1}(i_k'+i_k^*)=\mathca

Figures (5)

  • Figure 1: Performance of the compared algorithms. Left: softmax problem. Middle: Matrix game problem of size $(n,m)=(896,128)$. Right: Matrix game of size $(n,m)=(448,64)$.
  • Figure 2: Performance of the compared algorithms. Left: robustness test on diabetes dataset. Right: Long-run test on Boston housing dataset. Right: robustness test on the softmax problem
  • Figure 3: Performance of the compared algorithms in network training. Left: Training Loss. Right: Validation Loss. Right: Accuracy.
  • Figure 4: Performance of the compared algorithms on the softmax problem. From left to right: $\mu=0.1$, $\mu = 0.01$ and $\mu = 0.001$.
  • Figure 5: Performance of the compared algorithms on the $L_p$ norm problem. Left: $p = 1$ with diabetes. Middle: $p = 1.5$ with boston. Right: $p = 2$ with boston.

Theorems & Definitions (41)

  • Proposition 1
  • Remark 1
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Lemma 3
  • Theorem 2
  • Remark 2
  • Remark 3
  • Remark 4
  • ...and 31 more