Automated algorithm design via Nevanlinna-Pick interpolation

TL;DR

The paper tackles the challenge of designing optimization algorithms that approach fundamental performance limits by reframing algorithm construction as a frequency-domain control problem. Using Nevanlinna-Pick interpolation within a Lur'e-system formulation, it derives an automated synthesis procedure that yields the Interpolated Gradient Method (I-GM) with a tunable trade-off between per-iteration matrix-vector multiplications $\ell$ and convergence rate. The method provides analytical guarantees for explicitness, optimality, and linear convergence, and extends to problems of the form $\min_x f(x)$ s.t. $Ex=q$ with $f$ $m$-strongly convex and $L$-Lipschitz. Experimental results on logistic-regression and compressed-sensing-like tasks demonstrate that I-GM can outperform existing single-loop methods and that increasing $\ell$ yields faster convergence, aligning with the theoretical rate bound $\rho^\star = \max\left(1-\tfrac{2}{\kappa_f+1}, \left(1-\tfrac{1}{\kappa_E}\right)^{\ell}\right)$.

Abstract

The synthesis of optimization algorithms typically follows a design-first-analyze-later approach, which often obscures fundamental performance limitations and hinders the systematic design of algorithms operating at the achievable theoretical boundaries. Recently, a framework based on frequency-domain techniques from robust control theory has emerged as a powerful tool for automating algorithm synthesis. By integrating the design and analysis stages, this framework enables the identification of fundamental performance limits. In this paper, we build on this framework and extend it to address algorithms for solving strongly convex problems with equality constraints. As a result, we obtain a new class of algorithms that offers sharp trade-off between number of matrix multiplication per iteration and convergence rate.

Figures (5)

• Figure 1: (Left) Lur'e system representing the optimization algorithms. (Right) Decomposition of the system on the left into two subsystems coupled by the nonlinear element.
• Figure 2: I-GM ($q=0$)
• Figure 3: I-GM
• Figure 4: Convergence of the single loop algorithms on Example 1. Left: $(\kappa_f, \kappa_E) = (10^2, 10^6)$; Right: $(\kappa_f, \kappa_E) =(10^4, 10^2)$. The solid lines show convergence of algorithms while the dashed-line $\rho^\star(\ell=1)$ shows the theoretical convergence rate \ref{['eq.theorem']} with $\ell=1$.
• Figure 5: Convergence of the single loop algorithms on Example 2. Left: $(\kappa_f, \kappa_E) = (10^2, 10^4)$; Right: $(\kappa_f, \kappa_E) =(10^5, 10^4)$. The solid lines show convergence of algorithms while the dashed-line $\rho^\star(\ell=1)$ shows the theoretical convergence rate \ref{['eq.theorem']} with $\ell=1$.