Progressively Sampled Equality-Constrained Optimization

TL;DR

The paper tackles nonconvex optimization with equality constraints defined by an expectation, focusing on large-scale problems where the constraint is an average over many terms. It proposes Progressive Constraint-Sampling Method (PCSM), which solves a sequence of subproblems on expanding constraint-term samples ${\cal S}$ to reduce per-iteration cost and improve worst-case sample complexity compared to solving the full-sample problem directly. Theoretical guarantees are developed using acute perturbation analysis and recent constrained optimization complexity results, with least-squares multipliers ensuring principled stationarity notions for constrained subproblems. Numerical experiments demonstrate practical gains, validating that progressively enlarging the constraint sample can yield efficient approximate stationary points for the full problem. The approach provides a principled framework for handling large-scale equality-constrained SAA problems in settings like physics-informed learning and large-scale least squares with known target residuals.

Abstract

An algorithm is proposed, analyzed, and tested for solving continuous nonlinear-equality-constrained optimization problems where the constraints are defined by an expectation or an average over a large (finite) number of terms. The main idea of the algorithm is to solve a sequence of equality-constrained problems, each involving a finite sample of constraint-function terms, over which the sample set grows progressively. Under assumptions about the constraint functions and their first- and second-order derivatives that are reasonable in some real-world settings of interest, it is shown that -- with a sufficiently large initial sample -- solving a sequence of problems defined through progressive sampling yields a better worst-case sample complexity bound compared to solving a single problem with a full set of samples. The results of numerical experiments with a set of test problems demonstrate that the proposed approach can be effective in practice.