Verification of First-Order Methods for Parametric Quadratic Optimization

TL;DR

This work tackles the problem of verifying finite-step convergence for first-order methods solving parametric convex quadratic programs. It introduces a modular framework that encodes proximal algorithms as affine and element-wise maximum steps and directly models warm-start effects through explicit initial-iterate and parameter sets. By proving NP-hardness for the general verification problem and developing strong SDP relaxations with bound propagation and triangle relaxations, the approach yields less pessimistic, absolute worst-case bounds compared with traditional convergence analyses and PEP-based methods. Across NNLS, network utility maximization, Lasso, and optimal control, the method demonstrates tighter bounds and practical convergence insights, illustrating its potential to inform real-time algorithm design and parameter selection.

Abstract

We introduce a numerical framework to verify the finite step convergence of first-order methods for parametric convex quadratic optimization. We formulate the verification problem as a mathematical optimization problem where we maximize a performance metric (e.g., fixed-point residual at the last iteration) subject to constraints representing proximal algorithm steps (e.g., linear system solutions, projections, or gradient steps). Our framework is highly modular because we encode a wide range of proximal algorithms as variations of two primitive steps: affine steps and element-wise maximum steps. Compared to standard convergence analysis and performance estimation techniques, we can explicitly quantify the effects of warm-starting by directly representing the sets where the initial iterates and parameters live. We show that the verification problem is NP-hard, and we construct strong semidefinite programming relaxations using various constraint tightening techniques. Numerical examples in nonnegative least squares, network utility maximization, Lasso, and optimal control show a significant reduction in pessimism of our framework compared to standard worst-case convergence analysis techniques.

Figures (12)

• Figure 1: Unconstrained QP experiment with different the initial iterate sets (dashed circles). On the left, we show the iterate paths, and on the right, we plot the worst cast fixed-point residuals for up to $K=10$ steps. We also solve the PEP SDP with initial distance to optimality bounded by 1 (solid black circle). The verification problem and PEP agree on the worst-case value for $Z_1$.
• Figure 2: Unconstrained QP example with two parameter sets and fixed initial iterate $z^0$. On the left, we show the iterate paths for the worst-case parameter values together with the contour plots of the worst-case functions. The shaded gray area represents the initial condition for the PEP problem, which is large enough to include the initial iterate $z^0$. On the right we plot the worst-case fixed-point residuals for $K=10$ steps.
• Figure 3: Unconstrained QP example with two different quadratic terms. On the left, we show the worst-case iterate paths for up to $K=10$ steps. On the right, we compare the worst-case residuals to those from the PEP SDP.
• Figure 4: Results for the strongly convex NNLS instance with $\mu=20, L=100$, for fixed step sizes. Full results are in Tables \ref{['tab:nnls_fixed_pt1']} and \ref{['tab:nnls_fixed_pt2']}.
• Figure 5: Results to show the effect of $t$ at $K=4$ for the same strongly convex instance as in Figure \ref{['fig:strong_NNLS_gridt']}. There is a discrepancy between the PEP and VPSDP bounds on the $t$ that leads to the smallest residual.

Theorems & Definitions (10)

• Proposition 3.1
• Theorem 4.1
• Proposition 4.1: objective reformulation
• Proposition 4.2: affine step
• Proposition 4.3: element-wise maximum step
• Proposition 4.4: hypercubes
• Proposition 4.5: polyhedra
• Proposition 4.6: $\ell_2$-ball
• Lemma B.1: trace of positive semidefinite matrix product
• proof