Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Non-Stationary Stochastic Quasar-Convex Optimization

Yuen-Man Pun, Iman Shames

TL;DR

This paper develops a unified online optimization framework for time-varying losses that satisfy quasar-convexity and strong quasar-convexity, deriving dynamic regret bounds for online gradient descent in terms of path variation and gradient noise. It extends the theory to generalized linear models with time-varying parameters and analyzes activation functions including Leaky ReLU, logistic, and ReLU, providing activation-specific constants and step-size guidelines. The results yield sublinear regret under sublinear variation and noise growth, and are complemented by simulations demonstrating sublinear regret growth even with noise. The findings offer a principled approach for adapting non-convex unimodal objectives in online settings and have implications for real-time neural-model-like learning and dynamic system identification.

Abstract

Recent research has shown that quasar-convexity can be found in applications such as identification of linear dynamical systems and generalized linear models. Such observations have in turn spurred exciting developments in design and analysis algorithms that exploit quasar-convexity. In this work, we study the online stochastic quasar-convex optimization problems in a dynamic environment. We establish regret bounds of online gradient descent in terms of cumulative path variation and cumulative gradient variance for losses satisfying quasar-convexity and strong quasar-convexity. We then apply the results to generalized linear models (GLM) when the underlying parameter is time-varying. We establish regret bounds of online gradient descent when applying to GLMs with leaky ReLU activation function, logistic activation function, and ReLU activation function. Numerical results are presented to corroborate our findings.

Online Non-Stationary Stochastic Quasar-Convex Optimization

TL;DR

This paper develops a unified online optimization framework for time-varying losses that satisfy quasar-convexity and strong quasar-convexity, deriving dynamic regret bounds for online gradient descent in terms of path variation and gradient noise. It extends the theory to generalized linear models with time-varying parameters and analyzes activation functions including Leaky ReLU, logistic, and ReLU, providing activation-specific constants and step-size guidelines. The results yield sublinear regret under sublinear variation and noise growth, and are complemented by simulations demonstrating sublinear regret growth even with noise. The findings offer a principled approach for adapting non-convex unimodal objectives in online settings and have implications for real-time neural-model-like learning and dynamic system identification.

Abstract

Recent research has shown that quasar-convexity can be found in applications such as identification of linear dynamical systems and generalized linear models. Such observations have in turn spurred exciting developments in design and analysis algorithms that exploit quasar-convexity. In this work, we study the online stochastic quasar-convex optimization problems in a dynamic environment. We establish regret bounds of online gradient descent in terms of cumulative path variation and cumulative gradient variance for losses satisfying quasar-convexity and strong quasar-convexity. We then apply the results to generalized linear models (GLM) when the underlying parameter is time-varying. We establish regret bounds of online gradient descent when applying to GLMs with leaky ReLU activation function, logistic activation function, and ReLU activation function. Numerical results are presented to corroborate our findings.
Paper Structure (15 sections, 15 theorems, 94 equations, 3 figures, 1 table)

This paper contains 15 sections, 15 theorems, 94 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Statement and Preliminaries
  3. Online Quasar-Convex Optimization
  4. Online Strongly Quasar-Convex Optimization
  5. Applications
  6. Leaky ReLU
  7. Logistic Function
  8. Single ReLU
  9. Simulations
  10. Conclusion
  11. Proofs for Theorem \ref{['thm:str-qua-cvx']}
  12. Proofs for Section \ref{['sec:GLM']}
  13. Proofs for Section \ref{['subsec:LeakyReLU']}
  14. Proof for Section \ref{['subsec:logistic']}
  15. Proofs for Section \ref{['subsec:relu']}

Key Result

Proposition 1

Let $f\colon\mathbb{R}^n\to\mathbb{R}$ be a differentiable function. If $f$ is $L$-smooth; i.e., for all $\bm{w},\bm{w}'\in\mathbb{R}^n$, then $f$ is $\Gamma$-weakly smooth where $\Gamma = 2L$.

Figures (3)

  • Figure 1: Leaky ReLU
  • Figure 2: Logistic
  • Figure 3: ReLU

Theorems & Definitions (26)

  • Definition 1: Quasar-Convexity HSS20
  • Definition 2: Weak Smoothness HMR18
  • Proposition 1: Smoothness Implies Weak Smoothness
  • proof
  • Theorem 1: Regret Bounds for Quasar-Convex Losses
  • Remark 1: Boundedness of Sets of Interest
  • Remark 2: Step Size Selection for Quasar-Convex Losses
  • proof
  • Proposition 2: Convergence of Offline Gradient Descent
  • Remark 3: Step Size Selection for Strongly Quasar-Convex Losses
  • ...and 16 more