Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Exponential Averaging Process with Strong Convergence Properties

Frederik Köhne, Anton Schiela

TL;DR

This work addresses the limitation of exponential moving averages (EMA) in providing strong stochastic convergence when observations are taken along trajectories of random dynamical systems. It introduces p-EMA, a time-dependent smoothing with weights decaying as $\gamma_n=1-1/(n+1)^p$ for $p\in(\tfrac{1}{2},1]$, and develops an averaging-scheme framework to establish almost-sure convergence under summable decay of correlations. The paper proves convergence guarantees for p-EMA, connects these results to SGD dynamics via invariant measures, and shows how p-EMA can improve adaptive step-size estimators in stochastic optimization, supported by numerical studies. The findings offer a principled smoothing approach with strong theoretical guarantees that enhances online estimation and adaptive algorithms in stochastic, potentially non-stationary, environments.

Abstract

Averaging, or smoothing, is a fundamental approach to obtain stable, de-noised estimates from noisy observations. In certain scenarios, observations made along trajectories of random dynamical systems are of particular interest. One popular smoothing technique for such a scenario is exponential moving averaging (EMA), which assigns observations a weight that decreases exponentially in their age, thus giving younger observations a larger weight. However, EMA fails to enjoy strong stochastic convergence properties, which stems from the fact that the weight assigned to the youngest observation is constant over time, preventing the noise in the averaged quantity from decreasing to zero. In this work, we consider an adaptation to EMA, which we call $p$-EMA, where the weights assigned to the last observations decrease to zero at a subharmonic rate. We provide stochastic convergence guarantees for this kind of averaging under mild assumptions on the autocorrelations of the underlying random dynamical system. We further discuss the implications of our results for a recently introduced adaptive step size control for Stochastic Gradient Descent (SGD), which uses $p$-EMA for averaging noisy observations.

An Exponential Averaging Process with Strong Convergence Properties

TL;DR

This work addresses the limitation of exponential moving averages (EMA) in providing strong stochastic convergence when observations are taken along trajectories of random dynamical systems. It introduces p-EMA, a time-dependent smoothing with weights decaying as for , and develops an averaging-scheme framework to establish almost-sure convergence under summable decay of correlations. The paper proves convergence guarantees for p-EMA, connects these results to SGD dynamics via invariant measures, and shows how p-EMA can improve adaptive step-size estimators in stochastic optimization, supported by numerical studies. The findings offer a principled smoothing approach with strong theoretical guarantees that enhances online estimation and adaptive algorithms in stochastic, potentially non-stationary, environments.

Abstract

Averaging, or smoothing, is a fundamental approach to obtain stable, de-noised estimates from noisy observations. In certain scenarios, observations made along trajectories of random dynamical systems are of particular interest. One popular smoothing technique for such a scenario is exponential moving averaging (EMA), which assigns observations a weight that decreases exponentially in their age, thus giving younger observations a larger weight. However, EMA fails to enjoy strong stochastic convergence properties, which stems from the fact that the weight assigned to the youngest observation is constant over time, preventing the noise in the averaged quantity from decreasing to zero. In this work, we consider an adaptation to EMA, which we call -EMA, where the weights assigned to the last observations decrease to zero at a subharmonic rate. We provide stochastic convergence guarantees for this kind of averaging under mild assumptions on the autocorrelations of the underlying random dynamical system. We further discuss the implications of our results for a recently introduced adaptive step size control for Stochastic Gradient Descent (SGD), which uses -EMA for averaging noisy observations.
Paper Structure (22 sections, 9 theorems, 101 equations, 7 figures)

This paper contains 22 sections, 9 theorems, 101 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Convergence of p-EMA
  3. Averaging Schemes
  4. p-EMA Induces an Averaging Scheme
  5. Autocorrelations
  6. On the Condition p
  7. The case p > 1
  8. The case p < 1/2
  9. SGD and Invariant Measures
  10. Stochastic Gradient Descent
  11. Interpolating vs. Non-Interpolating Setting
  12. Invariant Measures
  13. Implications on Adaptive Step Size Estimators
  14. Adaptive Step Sizes
  15. Exploiting Convergence of p-EMA
  16. ...and 7 more sections

Key Result

theorem 1

Consider a sequence of non-negative random variables $X_n$. Let $\{b_n\}$ be a sequence of positive numbers. Suppose that the following conditions hold: $A_n \to \infty$ as $n \to \infty$, for all sufficiently large $n-m$, where $C$ is a constant, and for some function $\psi \in \Psi_c$. Then

Figures (7)

  • Figure 1.1: Behavior of the weights in the different averaging procedures.
  • Figure 3.1: Comparison of weights for $p$-EMA with $p$ outside the admissible interval $(\frac{1}{2}, 1]$.
  • Figure 6.1: Comparison of different averaging schemes on asymptotically stationary data.
  • Figure 6.2: Comparison of different averaging schemes on data generated by a jump process.
  • Figure 6.3: Convergence of suggested step size: Artificial Problem.
  • ...and 2 more figures

Theorems & Definitions (22)

  • definition 1
  • definition 2
  • remark 1
  • theorem 1: Korchevsky2010
  • theorem 2
  • proof
  • lemma 1
  • proof
  • definition 3
  • proposition 1
  • ...and 12 more