Small-Disturbance Input-to-State Stability of Perturbed Gradient Flows: Applications to LQR Problem
Leilei Cui, Zhong-Ping Jiang, Eduardo D. Sontag
TL;DR
The paper addresses robustness of gradient-flow optimization under perturbations by establishing small-disturbance ISS for perturbed gradient flows under coercivity and a CJS-PL gradient-dominance condition. It develops a Lyapunov characterization of SSD, proves sufficiency (and discussion of necessity) for the ISS property, and applies the theory to LQR policy optimization, showing SSD for standard, natural, and Newton gradient flows. The LQR analysis demonstrates that the loss J2(K) is coercive with a gradient-dominance bound, enabling explicit ISS-Lyapunov constructions and robust convergence bounds under gradient estimation or numerical perturbations. This work provides a principled framework to quantify robustness of data-driven control and RL methods that rely on gradient-based policy updates, with concrete guarantees on convergence to neighborhoods of the optimum under bounded disturbances.
Abstract
This paper studies the effect of perturbations on the gradient flow of a general nonlinear programming problem, where the perturbation may arise from inaccurate gradient estimation in the setting of data-driven optimization. Under suitable conditions on the objective function, the perturbed gradient flow is shown to be small-disturbance input-to-state stable (ISS), which implies that, in the presence of a small-enough perturbation, the trajectories of the perturbed gradient flow must eventually enter a small neighborhood of the optimum. This work was motivated by the question of robustness of direct methods for the linear quadratic regulator problem, and specifically the analysis of the effect of perturbations caused by gradient estimation or round-off errors in policy optimization. We show small-disturbance ISS for three of the most common optimization algorithms: standard gradient flow, natural gradient flow, and Newton gradient flow.