An overview of the fractional-order gradient descent method and its applications

TL;DR

This paper surveys fractional-gradient descent approaches and demonstrates that standard generalizations, which replace the gradient with fractional derivatives, often fail to guarantee convergence to the true extremum. It advocates the Fractional Continuous Time Method (FCTM), which replaces the time derivative with a Caputo fractional derivative to ensure that equilibrium points coincide with extrema, with proven convergence to the extremum for $0<\alpha<1$ and Mittag-Leffler–type stability. Through applications to Vandermonde interpolation and the Thomson problem, the authors show that carefully chosen $\alpha$ can accelerate convergence or reduce residuals compared with classical gradient descent, albeit with higher computational cost due to nonlocal fractional dynamics. The work highlights the potential of FCTM for complex optimization tasks in chemical and physical contexts while acknowledging the need for deeper theoretical results for $\alpha\ge 1$ and practical guidelines for selecting the fractional order.

Abstract

Recent studies have shown that fractional calculus is an effective alternative mathematical tool in various scientific fields. However, some investigations indicate that results established in differential and integral calculus do not necessarily hold true in fractional calculus. In this work we will compare various methods presented in the literature to improve the Gradient Descent Method, in terms of convergence of the method, convergence to the extreme point, and convergence rate. In general, these methods that generalize the gradient descent algorithm by replacing the gradient with a fractional-order operator are inefficient in achieving convergence to the extremum point of the objective function. To avoid these difficulties, we proposed to choose the Fractional Continuous Time algorithm to generalize the gradient method. In this approach, the convergence of the method to the extreme point of the function is guaranteed by introducing the fractional order in the time derivative, rather than in of the gradient. In this case, the issue of finding the extreme point is resolved, while the issue of stability at the equilibrium point remains. Fractional Continuous Time method converges to extreme point of cost function when fractional-order is between 0 and 1. The simulations shown in this work suggests that a similar result can be found when $1 \leq α\leq 2$. { This paper highlights the main advantages and disadvantages of generalizations of the gradient method using fractional derivatives, aiming to optimize convergence in complex problems. Some chemical problems, with n=11 and 24 optimization parameters, are employed as means of evaluating the efficacy of the propose algorithms. In general, previous studies are restricted to mathematical questions and simple illustrative examples.

Figures (8)

• Figure 1: The optimal value obtained at each iteration, using Gradient Descent Method (FGDM) (thick solid line). The solutions obtained by GDM-1 4P and (FGDM-2) 5P, both with $\alpha=0.9$, is represented by the dotted line and dashed line, respectively. The thin continuous line shows the result found by Fractional Continuous Time Method (FCTM), with $\alpha=0.9$. The expected result is 3 ($c=3$).
• Figure 2: The solutions obtained by Fractional Continuous Time Method (FCTM), with different values of fractional order $\alpha$, for the problem of minimizing the function $f(u)=(u-3)^2$.
• Figure 3: The performance of the Fractional Continuous Time Method, with $1\leq\alpha\leq 2$ and different initial conditions: for all $u(0)=1$ and $u^\prime(0)=0$ (thin line) or $0.5$ (thick line). Result obtained with $\alpha=1.2$ (dotted line), $\alpha=1.5$ (dashed line), and $\alpha=1.7$ (continuous line).
• Figure 4: The error $V(u)=(u(t)-u^*)^2$ as a function of time. This result is obtained by Fractional Continuous Time Method (FCTM), for the problem of minimizing the function $f(u)=(u-3)^2$.
• Figure 5: The performance of the Fractional Continuous Time Method to solve the interpolation polynomial problem.