Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Simple yet Highly Accurate Prediction-Correction Algorithm for Time-Varying Optimization

Tomoya Kamijima, Naoki Marumo, Akiko Takeda

TL;DR

SHARP introduces a Hessian-free prediction-correction framework for unconstrained time-varying optimization by extrapolating past solutions via Lagrange interpolation and correcting with a simple descent step. It achieves $O(h^{p})$ asymptotic tracking error under bounded $p$th derivatives of the target trajectory and local linear convergence of the correction, and it extends to non-convex objectives including PL functions. The method relies on an acceptance criterion to prevent divergence, ensuring stability even when the function is non-convex. Empirical results on toy, target-tracking, and non-convex regression problems demonstrate high tracking accuracy and robustness, with SHARP outperforming existing approaches in several settings. This work offers a computationally efficient alternative to Hessian-based predictions while delivering strong theoretical guarantees and practical performance.

Abstract

This paper proposes a simple yet highly accurate prediction-correction algorithm, SHARP, for unconstrained time-varying optimization problems. Its prediction is based on an extrapolation derived from the Lagrange interpolation of past solutions. Since this extrapolation can be computed without Hessian matrices or even gradients, the computational cost is low. To ensure the stability of the prediction, the algorithm includes an acceptance condition that rejects the prediction when the update is excessively large. The proposed method achieves a tracking error of $O(h^{p})$, where $h$ is the sampling period, assuming that the $p$th derivative of the target trajectory is bounded and the convergence of the correction step is locally linear. We also prove that the method can track a trajectory of stationary points even if the objective function is non-convex. Numerical experiments demonstrate the high accuracy of the proposed algorithm.

A Simple yet Highly Accurate Prediction-Correction Algorithm for Time-Varying Optimization

TL;DR

SHARP introduces a Hessian-free prediction-correction framework for unconstrained time-varying optimization by extrapolating past solutions via Lagrange interpolation and correcting with a simple descent step. It achieves asymptotic tracking error under bounded th derivatives of the target trajectory and local linear convergence of the correction, and it extends to non-convex objectives including PL functions. The method relies on an acceptance criterion to prevent divergence, ensuring stability even when the function is non-convex. Empirical results on toy, target-tracking, and non-convex regression problems demonstrate high tracking accuracy and robustness, with SHARP outperforming existing approaches in several settings. This work offers a computationally efficient alternative to Hessian-based predictions while delivering strong theoretical guarantees and practical performance.

Abstract

This paper proposes a simple yet highly accurate prediction-correction algorithm, SHARP, for unconstrained time-varying optimization problems. Its prediction is based on an extrapolation derived from the Lagrange interpolation of past solutions. Since this extrapolation can be computed without Hessian matrices or even gradients, the computational cost is low. To ensure the stability of the prediction, the algorithm includes an acceptance condition that rejects the prediction when the update is excessively large. The proposed method achieves a tracking error of , where is the sampling period, assuming that the th derivative of the target trajectory is bounded and the convergence of the correction step is locally linear. We also prove that the method can track a trajectory of stationary points even if the objective function is non-convex. Numerical experiments demonstrate the high accuracy of the proposed algorithm.

Paper Structure

This paper contains 26 sections, 13 theorems, 85 equations, 11 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related Work
  4. Prediction-correction algorithms for strongly convex functions
  5. Non-convex time-varying optimization
  6. Extrapolation-based prediction-correction algorithms
  7. Notation
  8. Proposed Method
  9. General Tracking Error Analysis
  10. Assumptions
  11. Acceptance of Higher-Order Predictions
  12. General Tracking Error
  13. Specific Tracking Error Analysis
  14. Case 1: Strongly Convex Functions
  15. Case 2: PŁ Functions
  16. ...and 11 more sections

Key Result

Lemma 2.1

Let $0\leq\underline k<\overline k\leq\infty$ be constants. Suppose that $\bm x^\ast\colon[t_{\underline k},t_{\overline k}]\to\mathbb{R}^n$ is $p$-times differentiable with $1\leq p\leq\overline k-\underline k$. Then, the following holds for all $\underline k+p\leq k\leq\overline k$:

Figures (11)

  • Figure 1: The prediction $\hat{\bm x}_k^4$ is computed from the Lagrange interpolation of $\bm x_{k-4},\ldots,\bm x_{k-1}$, which approximates $\bm x^\ast(t_k)$ well.
  • Figure 2: The prediction $\hat{\bm x}_k^2$ is close to the next target point $\bm x^\ast(t_{k})$.
  • Figure 3: The prediction $\hat{\bm x}_k^2$ is far from both $\bm x^\ast(t_{k})$ and $\tilde{\bm x}^\ast(t_{k})$.
  • Figure 5: $h=1$
  • Figure 6: $h=0.1$
  • ...and 6 more figures

Theorems & Definitions (30)

  • Lemma 2.1
  • Lemma 3.2
  • proof
  • Proposition 3.3
  • proof
  • Remark 3.4
  • Lemma 3.5
  • proof
  • Theorem 3.6: Tracking Error of SHARP
  • proof
  • ...and 20 more