Projected Gradient Methods with Momentum

TL;DR

This work tackles constrained nonlinear optimization with a smooth but potentially nonconvex objective $f$ over a convex feasible set $\mathcal{S}$, and develops a momentum-augmented projected gradient framework (PGMM). The key idea is to form search directions $d_k = \alpha_k \hat d_k + \beta_k \hat s_k$ where $\hat d_k = \mathcal{P}_{\mathcal{S}}[x_k - \eta_k \nabla f(x_k)] - x_k$ and $\hat s_k = \mathcal{P}_{\mathcal{S}}[x_k+(x_k-x_{k-1})]-x_k$, with $\eta_k$ in a bounded interval and $\alpha_k, \beta_k$ chosen by solving a closed-form two-dimensional quadratic subproblem (potentially refined via a $2\times2$ curvature matrix $H_k$). The method updates via $x_{k+1}=x_k+\mu_k d_k$ with Armijo line search, and the authors prove global convergence and an $\mathcal{O}(\varepsilon^{-2})$ complexity bound for nonconvex problems. Empirical results on $\ell_1$-ball and box-constrained problems show that PGMM outperforms the standard spectral projected gradient, illustrating the practical value of adding momentum in constrained settings. Theoretical contributions include a general line-search framework for constrained optimization with gradient-related directions and a concrete, efficiently computable momentum variant with guaranteed descent properties. Overall, the paper demonstrates that momentum can be effectively integrated into projection-based methods with provable guarantees and tangible speedups.

Abstract

We focus on the optimization problem with smooth, possibly nonconvex objectives and a convex constraint set for which the Euclidean projection operation is practically available. Focusing on this setting, we carry out a general convergence and complexity analysis for algorithmic frameworks. Consequently, we discuss theoretically sound strategies to integrate momentum information within classical projected gradient type algorithms. One of these approaches is then developed in detail, up to the definition of a tailored algorithm with both theoretical guarantees and reasonable per-iteration cost. The proposed method is finally shown to outperform the standard (spectral) projected gradient method in two different experimental benchmarks, indicating that the addition of momentum terms is as beneficial in the constrained setting as it is in the unconstrained scenario.