On the Convergence of the Gradient Descent Method with Stochastic Fixed-point Rounding Errors under the Polyak-Lojasiewicz Inequality
Lu Xia, Michiel E. Hochstenbach, Stefano Massei
TL;DR
This work analyzes gradient descent under fixed-point arithmetic with rounding errors for functions satisfying the Polyak–Łojasiewicz inequality. It shows that unbiased SR preserves linear convergence, while ε-biased SR_ε yields stricter convergence bounds and faster practical convergence; the authors derive a three-case update framework to capture updating behavior across regimes where gradient magnitudes compare to the rounding precision. Theoretical results are complemented by simulations on quadratic problems, Rosenbrock/Himmelblau functions, binary logistic regression, and a four-layer neural network, demonstrating that SR_ε often outperforms SR and RN, and that fixed-point rounding can be competitive with floating-point when using appropriate bias. The findings provide actionable guidance for low-precision training, highlighting when to employ SR versus SR_ε to mitigate stagnation and accelerate convergence in real-world learning tasks.
Abstract
When training neural networks with low-precision computation, rounding errors often cause stagnation or are detrimental to the convergence of the optimizers; in this paper we study the influence of rounding errors on the convergence of the gradient descent method for problems satisfying the Polyak-\Lojasiewicz inequality. Within this context, we show that, in contrast, biased stochastic rounding errors may be beneficial since choosing a proper rounding strategy eliminates the vanishing gradient problem and forces the rounding bias in a descent direction. Furthermore, we obtain a bound on the convergence rate that is stricter than the one achieved by unbiased stochastic rounding. The theoretical analysis is validated by comparing the performances of various rounding strategies when optimizing several examples using low-precision fixed-point number formats.