Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems

Fedor Stonyakin, Alexander Titov, Mohammad Alkousa, Oleg Savchuk, Alexander Gasnikov

Abstract

Recently there were proposed some innovative convex optimization concepts, namely, relative smoothness [1] and relative strong convexity [2,3]. These approaches have significantly expanded the class of applicability of gradient-type methods with optimal estimates of the convergence rate, which are invariant regardless of the dimensionality of the problem. Later Yu. Nesterov and H. Lu introduced some modifications of the Mirror Descent method for convex minimization problems with the corresponding analogue of the Lipschitz condition (so-called relative Lipschitz continuity). By introducing an artificial inaccuracy to the optimization model, we propose adaptive methods for minimizing a convex Lipschitz continuous function, as well as for the corresponding class of variational inequalities. We also consider an adaptive "universal" method, applicable to convex minimization problems both on the class of relatively smooth and relatively Lipschitz continuous functionals with optimal estimates of the convergence rate. The universality of the method makes it possible to justify the applicability of the obtained theoretical results to a wider class of convex optimization problems. We also present the results of numerical experiments.

Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems

Abstract

Recently there were proposed some innovative convex optimization concepts, namely, relative smoothness [1] and relative strong convexity [2,3]. These approaches have significantly expanded the class of applicability of gradient-type methods with optimal estimates of the convergence rate, which are invariant regardless of the dimensionality of the problem. Later Yu. Nesterov and H. Lu introduced some modifications of the Mirror Descent method for convex minimization problems with the corresponding analogue of the Lipschitz condition (so-called relative Lipschitz continuity). By introducing an artificial inaccuracy to the optimization model, we propose adaptive methods for minimizing a convex Lipschitz continuous function, as well as for the corresponding class of variational inequalities. We also consider an adaptive "universal" method, applicable to convex minimization problems both on the class of relatively smooth and relatively Lipschitz continuous functionals with optimal estimates of the convergence rate. The universality of the method makes it possible to justify the applicability of the obtained theoretical results to a wider class of convex optimization problems. We also present the results of numerical experiments.

Paper Structure

This paper contains 9 sections, 6 theorems, 85 equations, 4 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Basic definitions and notations
  3. Adaptive Method for Variational Inequalities with Relatively Bounded Operators
  4. Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems
  5. Universal Algorithms for Relatively Smooth and Relatively Lipschitz Continuous Convex Optimization Problems
  6. Numerical Experiments
  7. The Intersection of Ellipsoids Problem (IEP)
  8. Comparison with AdaMirr
  9. Support Vector Machine (SVM) and Inequality-Type Function Constraints

Key Result

Theorem \oldthetheorem

Let $g: Q\longrightarrow \mathbb {R}^n$ be a relatively bounded and monotone operator, i.e. Rel_Boud and monotone_operator hold, $L_0\leqslant \frac{2M^2}{\varepsilon}$. Then after the stopping of Algorithm adaptive_alg3, the following inequality holds Moreover, the total number of iterations will not exceed $N=\left\lceil\frac{4M^2R^2}{\varepsilon^2}\right\rceil.$

Figures (4)

  • Figure 1: The results of Algorithm \ref{['Alg_5']} for IEP with different values of $n$ and $m = 10$.
  • Figure 2: The results of Algorithms \ref{['adaptive_alg4']}, \ref{['Algor2']} and \ref{['Alg_5']} for IEP with $n=1000$ and $m = 10$.
  • Figure 3: The results of comparison of Algorithm \ref{['Alg_5']} and AdaMirr for IEP with $n = 1000$ and $m=10$.
  • Figure 4: The results of Algorithm \ref{['adaptive_alg3']}, for the problem \ref{['problem_min_for_SPP']}.

Theorems & Definitions (23)

  • Definition \oldthetheorem
  • Definition \oldthetheorem
  • Theorem \oldthetheorem
  • proof
  • Remark \oldthetheorem
  • Theorem \oldthetheorem
  • proof
  • Remark \oldthetheorem
  • Remark \oldthetheorem
  • Theorem \oldthetheorem
  • ...and 13 more