Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finite-time analysis of Multi-timescale Stochastic Optimization Algorithms

Kaustubh Kartikey, Shalabh Bhatnagar

Abstract

We present a finite-time analysis of two smoothed functional stochastic approximation algorithms for simulation-based optimization. The first is a two time-scale gradient-based method, while the second is a three time-scale Newton-based algorithm that estimates both the gradient and the Hessian of the objective function $J$. Both algorithms involve zeroth order estimates for the gradient/Hessian. Although the asymptotic convergence of these algorithms has been established in prior work, finite-time guarantees of two-timescale stochastic optimization algorithms in zeroth order settings have not been provided previously. For our Newton algorithm, we derive mean-squared error bounds for the Hessian estimator and establish a finite-time bound on $\min\limits_{0 \le m \le T} \mathbb{E}\| \nabla J(θ(m)) \|^2$, showing convergence to first-order stationary points. The analysis explicitly characterizes the interaction between multiple time-scales and the propagation of estimation errors. We further identify step-size choices that balance dominant error terms and achieve near-optimal convergence rates. We also provide corresponding finite-time guarantees for the gradient algorithm under the same framework. The theoretical results are further validated through experiments on the Continuous Mountain Car environment.

Finite-time analysis of Multi-timescale Stochastic Optimization Algorithms

Abstract

We present a finite-time analysis of two smoothed functional stochastic approximation algorithms for simulation-based optimization. The first is a two time-scale gradient-based method, while the second is a three time-scale Newton-based algorithm that estimates both the gradient and the Hessian of the objective function . Both algorithms involve zeroth order estimates for the gradient/Hessian. Although the asymptotic convergence of these algorithms has been established in prior work, finite-time guarantees of two-timescale stochastic optimization algorithms in zeroth order settings have not been provided previously. For our Newton algorithm, we derive mean-squared error bounds for the Hessian estimator and establish a finite-time bound on , showing convergence to first-order stationary points. The analysis explicitly characterizes the interaction between multiple time-scales and the propagation of estimation errors. We further identify step-size choices that balance dominant error terms and achieve near-optimal convergence rates. We also provide corresponding finite-time guarantees for the gradient algorithm under the same framework. The theoretical results are further validated through experiments on the Continuous Mountain Car environment.

Paper Structure

This paper contains 38 sections, 6 theorems, 90 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. NSF1 and GSF1 Algorithms
  4. Problem Setup
  5. Smoothed Functional Gradient and Hessian Estimation
  6. Recursive Hessian Tracking (Fastest Time-Scale)
  7. Gradient Tracking (Intermediate Time-Scale)
  8. Parameter Update (Slowest Time-Scale)
  9. The GSF1 Algorithm
  10. Assumptions
  11. Finite-time analysis of NSF1 and GSF1
  12. Finite-time analysis of NSF1
  13. Step 1: Splitting into diagonal and off-diagonal components
  14. Step 2: Recursion for the diagonal element error
  15. Step 3: Expansion into five components
  16. ...and 23 more sections

Key Result

Lemma 1

Under Assumptions a1 and a2, $\nabla J(\theta)$ and $\nabla^2 J(\theta)$ are both Lipschitz continuous on $C$.

Figures (1)

  • Figure 1: Convergence of algorithms for Mountain Car environment

Theorems & Definitions (9)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Remark 1
  • Theorem 1: Finite-Time Hessian Estimation Error
  • Theorem 2: Finite-Time Convergence to FOSP
  • Theorem 3: Finite-Time Gradient Estimation Error
  • Theorem 4: Finite-Time Convergence to FOSP