Incremental Gauss--Newton Methods with Superlinear Convergence Rates
Zhiling Zhou, Zhuanghua Liu, Chengchang Liu, Luo Luo
TL;DR
This work addresses solving systems ${\bf f}({\bf x})={\bf 0}$ with a Jacobian that is Hölder continuous of exponent $\nu\in(0,1]$ and a nondegenerate solution ${\bf x}^*$. It introduces Incremental Gauss--Newton (IGN), which updates one component per iteration and maintains aggregated estimates ${\bf H}^t$ and ${\bf G}^t={\bf H}^t^{-1}$ to achieve explicit local superlinear convergence, plus a mini-batch extension MB-IGN for faster rates. The authors prove convergence guarantees: $\|{\bf x}^{t+1}-{f x}^*\|$ decays at rate $r_{t+1} \le c^{(1+\nu)^{\lfloor t/n\rfloor-1}} r_t$ with a problem-dependent constant $c$, and in the Lipschitz-Jacobian case ($\nu=1$) obtain $n$-step quadratic convergence; MB-IGN generalizes this with batch size $k$ and $m=\lceil n/k\rceil$, yielding analogous rates. Numerical experiments on regularized logistic regression, Chandrasekhar's H-equation, and soft maximum minimization demonstrate that IGN/MB-IGN achieve faster convergence and lower computation time than several baselines, validating the practical benefit of incremental Newton-type methods for large-scale nonlinear equations.
Abstract
This paper addresses the challenge of solving large-scale nonlinear equations with Hölder continuous Jacobians. We introduce a novel Incremental Gauss--Newton (IGN) method within explicit superlinear convergence rate, which outperforms existing methods that only achieve linear convergence rate. In particular, we formulate our problem by the nonlinear least squares with finite-sum structure, and our method incrementally iterates with the information of one component in each round. We also provide a mini-batch extension to our IGN method that obtains an even faster superlinear convergence rate. Furthermore, we conduct numerical experiments to show the advantages of the proposed methods.