Prediction-Correction Algorithm for Time-Varying Smooth Non-Convex Optimization

TL;DR

A new prediction-correction algorithm that is applicable to large-scale and general non-convex problems and that is more accurate than TVGD is proposed and shown to be able to reduce the convergence error as the theoretical analyses suggest and outperform the existing algorithms.

Abstract

Time-varying optimization problems are prevalent in various engineering fields, and the ability to solve them accurately in real-time is becoming increasingly important. The prediction-correction algorithms used in smooth time-varying optimization can achieve better accuracy than that of the time-varying gradient descent (TVGD) algorithm. However, none of the existing prediction-correction algorithms can be applied to general non-strongly-convex functions, and most of them are not computationally efficient enough to solve large-scale problems. Here, we propose a new prediction-correction algorithm that is applicable to large-scale and general non-convex problems and that is more accurate than TVGD. Furthermore, we present convergence analyses of the TVGD and proposed prediction-correction algorithms for non-strongly-convex functions for the first time. In numerical experiments using synthetic and real datasets, the proposed algorithm is shown to be able to reduce the convergence error as the theoretical analyses suggest and outperform the existing algorithms.

Figures (20)

• Figure 1: Graph of $f(x) = \frac{x^2}{20} + \sin(x)$.
• Figure 2: Plots of the iterate $x_{k \mid k-1} - 10 t_k$ and gradient norm $\| \nabla_x f(x_{k \mid k-1};t_{k}) \|$ generated by the algorithms.
• Figure 3: Log plots of function value and gradient norm when $h = 1e^{-3}$.
• Figure 4: (Top) Log-log plots of maximum and mean of the gradient norm $\| \nabla_x f(x_{k \mid k-1};t_k) \|$ versus sampling period. (Bottom) Log-log plots of maximum and mean of the optimality gap $f(x_{k \mid k-1};t_k) - f_k^\ast$ versus sampling period. Maximum and mean are computed based on the results of the last half of the iterations.
• Figure 5: (Left) Graph of Geman-McClure loss function. (Right) Graph of Welsch loss function.

Theorems & Definitions (30)

• Lemma 3.1
• proof
• Theorem 3.1
• proof
• Theorem 3.2
• proof
• Definition 3.1
• Theorem 3.3
• proof
• Proposition 4.1