Towards Quantifying the Preconditioning Effect of Adam
Rudrajit Das, Naman Agarwal, Sujay Sanghavi, Inderjit S. Dhillon
TL;DR
This work quantitatively analyzes the preconditioning effect of Adam on ill-conditioned problems, starting with deterministic d-dimensional quadratics. It shows that Adam can reduce dependence on the Hessian condition number κ, achieving iteration complexities that scale as min(d, κ) in the diagonal case and more generally as min(d ar κ sqrt{d ar κ κ_diag}, ar κ κ_diag) , with improved behavior for diagonally dominant Hessians. However, for sufficiently non-diagonal Hessians, Adam can underperform GD, and it may not converge to zero function value in practice, a phenomenon supported by empirical results and fixed-point analysis. Extending to per-coordinate smoothness and a modified PL condition, the paper demonstrates initialization-dependent regimes where Adam outperforms GD, providing a nuanced, condition-number-aware picture of when adaptive methods offer genuine optimization gains.
Abstract
There is a notable dearth of results characterizing the preconditioning effect of Adam and showing how it may alleviate the curse of ill-conditioning -- an issue plaguing gradient descent (GD). In this work, we perform a detailed analysis of Adam's preconditioning effect for quadratic functions and quantify to what extent Adam can mitigate the dependence on the condition number of the Hessian. Our key finding is that Adam can suffer less from the condition number but at the expense of suffering a dimension-dependent quantity. Specifically, for a $d$-dimensional quadratic with a diagonal Hessian having condition number $κ$, we show that the effective condition number-like quantity controlling the iteration complexity of Adam without momentum is $\mathcal{O}(\min(d, κ))$. For a diagonally dominant Hessian, we obtain a bound of $\mathcal{O}(\min(d \sqrt{d κ}, κ))$ for the corresponding quantity. Thus, when $d < \mathcal{O}(κ^p)$ where $p = 1$ for a diagonal Hessian and $p = 1/3$ for a diagonally dominant Hessian, Adam can outperform GD (which has an $\mathcal{O}(κ)$ dependence). On the negative side, our results suggest that Adam can be worse than GD for a sufficiently non-diagonal Hessian even if $d \ll \mathcal{O}(κ^{1/3})$; we corroborate this with empirical evidence. Finally, we extend our analysis to functions satisfying per-coordinate Lipschitz smoothness and a modified version of the Polyak-Łojasiewicz condition.