More Optimal Fractional-Order Stochastic Gradient Descent for Non-Convex Optimization Problems

TL;DR

The paper introduces 2SEDFOSGD, a geometry-aware fractional-order stochastic gradient method that dynamically tunes the fractional exponent by leveraging Two-Scale Effective Dimension (2SED). By integrating 2SED with Caputo fractional updates, the approach aligns memory depth with local curvature via the Fisher Information, improving stability and convergence in non-convex optimization. Theoretical guarantees under standard smoothness and stochastic-gradient assumptions are provided, and empirical results on an autoregressive model under Gaussian and $\alpha$-stable noise demonstrate faster convergence and more robust parameter estimation than baseline FOSGD. This dimension-aware, adaptive memory mechanism offers a practical pathway to harness long-range gradient information in complex learning tasks.

Abstract

Fractional-order stochastic gradient descent (FOSGD) leverages fractional exponents to capture long-memory effects in optimization. However, its utility is often limited by the difficulty of tuning and stabilizing these exponents. We propose 2SED Fractional-Order Stochastic Gradient Descent (2SEDFOSGD), which integrates the Two-Scale Effective Dimension (2SED) algorithm with FOSGD to adapt the fractional exponent in a data-driven manner. By tracking model sensitivity and effective dimensionality, 2SEDFOSGD dynamically modulates the exponent to mitigate oscillations and hasten convergence. Theoretically, for onoconvex optimization problems, this approach preserves the advantages of fractional memory without the sluggish or unstable behavior observed in naïve fractional SGD. Empirical evaluations in Gaussian and $α$-stable noise scenarios using an autoregressive (AR) model highlight faster convergence and more robust parameter estimates compared to baseline methods, underscoring the potential of dimension-aware fractional techniques for advanced modeling and estimation tasks.

Figures (6)

• Figure 1: Under Gaussian noise at $\alpha=0.98$, both FOSGD and 2SEDFOSGD converge to the true AR parameters, with 2SEDFOSGD achieving faster and tighter estimates.
• Figure 2: Effective Fractional Order $\alpha_t$ and 2SED Under Gaussian Noise ($\alpha=0.98$). Over 1,400 iterations, $\alpha_t$ remains near 0.9793, and the 2SED measure fluctuates within 5.25--5.45, indicating stable adaptive performance.
• Figure 3: Under Gaussian noise at $\alpha=0.98$, 2SEDFOSGD maintains lower absolute estimation errors than FOSGD for both AR parameters.
• Figure 4: Under $\alpha$-stable noise with $\alpha_{\mathrm{stbl}}=1.8$, both FOSGD and 2SEDFOSGD converge near the true AR parameters, with 2SEDFOSGD showing smoother trajectories and fewer extreme deviations.
• Figure 5: Under $\alpha_{\mathrm{stbl}}=1.8$, FOSGD shows higher oscillations before stabilizing, while 2SEDFOSGD exhibits a smoother, consistently decreasing error trajectory for both $a_1$ and $a_2$.

Theorems & Definitions (13)

• Definition 1: Fisher Information datres2024two
• Definition 2: Empirical Fisher datres2024two
• Definition 3: Normalized Fisher Matrix datres2024two
• Definition 4: Two-Scale Effective Dimension datres2024two
• Definition 5: Caputo Derivative monje2010fractional
• Proposition 5.1: Bounded Iterates
• proof
• Lemma 1: Bounding the 2SED Measure
• proof
• Lemma 2: Descent Lemma for Non-Convex Case