A Simple First-Order Algorithm for Full-Rank Equality Constrained Optimization

TL;DR

This work introduces ADSWITCH, a simple objective-function-free first-order method for nonlinear optimization with equality constraints and full-rank Jacobians. The algorithm alternates tangential steps in the constraint tangent plane, computed via AdaGrad, with normal steps that reduce infeasibility, all without evaluating $f$ or using a merit function. A comprehensive worst-case complexity analysis yields a global convergence rate of $O(1/\sqrt{k})$, matching best-known rates for unconstrained first-order methods, and is complemented by numerical experiments on CUTEst problems showing competitive performance and notable robustness to gradient noise. The approach adapts trust-funnel ideas into a streamlined OFFO framework, offering practical reliability for noisy gradient evaluations and a foundation for future stochastic extensions and inequality-constrained extensions.

Abstract

A very simple first-order algorithm is proposed for solving nonlinear optimization problems with deterministic nonlinear equality constraints. This algorithm adaptively selects steps in the plane tangent to the constraints or steps that reduce infeasibility, without using a merit function or filter. The tangent steps are based on the AdaGrad method for unconstrained minimization. The objective function is never evaluated by the algorithm, making it suitable for noisy problems. Its worst-case evaluation complexity is analyzed, yielding a global convergence rate in O(1/sqrt{k}), which matches the best known rate of first-order methods for unconstrained problems. Numerical experiments are presented suggesting that the performance of the algorithm is comparable to that of first-order methods for unconstrained problems, and that its reliability is remarkably stable in the presence of noise on the gradient.

Figures (5)

• Figure 1: Evolution of the optimality measure (left: BYRDSPHR, right: ORTHREGA)
• Figure 2: Evolution of the optimality measure (left: LUKVLE2, right: BT1)
• Figure 3: Evolution of the optimality measure ( HS50)
• Figure 4: Reliability of the ADSWITCH algorithm in the presence of relative Gaussian noise on the gradient.
• Figure 5: Evolution of the two components of the optimality measure for a run of the noisy BT1 problem, with 5% (top left), 15% (top right), 25% (bottom left) and 50% (bottom right) relative Gaussian noise on the gradient

Theorems & Definitions (9)

• Lemma 3.1
• Lemma 3.2
• Lemma 3.3
• Lemma 3.4
• Lemma 3.5
• Lemma 3.6
• Lemma 3.7
• Lemma 3.8
• Theorem 3.9