Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proximal Gradient Dynamics and Feedback Control for Equality-Constrained Composite Optimization

Veronica Centorrino, Francesca Rossi, Francesco Bullo, Giovanni Russo

TL;DR

The paper addresses equality-constrained non-smooth composite optimization by introducing the proportional--integral proximal gradient dynamics (PI--PGD), a continuous-time closed-loop system with primal variables and Lagrange multipliers as control inputs. It proves the equivalence between stationary points of the optimization problem and equilibria of PI--PGD, and provides a contraction-based convergence analysis for affine equality constraints, yielding linear-exponential convergence to the equilibrium. The authors validate the approach on equality-constrained LASSO and nonlinear equality constraints, demonstrating robust performance and practical viability beyond strict assumptions. This work connects optimization with control-theoretic feedback design, offering a principled framework for solving constrained, possibly non-smooth problems with provable convergence properties and broad applicability in engineering and machine learning contexts.

Abstract

This paper studies equality-constrained composite minimization problems. This class of problems, capturing regularization terms and inequality constraints, naturally arises in a wide range of engineering and machine learning applications. To tackle these optimization problems, inspired by recent results, we introduce the \emph{proportional--integral proximal gradient dynamics} (PI--PGD): a closed-loop system where the Lagrange multipliers are control inputs and states are the problem decision variables. First, we establish the equivalence between the stationary points of the minimization problem and the equilibria of the PI--PGD. Then for the case of affine constraints, by leveraging tools from contraction theory we give a comprehensive convergence analysis for the dynamics, showing linear--exponential convergence towards the equilibrium. That is, the distance between each solution and the equilibrium is upper bounded by a function that first decreases linearly and then exponentially. Our findings are illustrated numerically on a set of representative examples, which include an exploratory application to nonlinear equality constraints.

Proximal Gradient Dynamics and Feedback Control for Equality-Constrained Composite Optimization

TL;DR

The paper addresses equality-constrained non-smooth composite optimization by introducing the proportional--integral proximal gradient dynamics (PI--PGD), a continuous-time closed-loop system with primal variables and Lagrange multipliers as control inputs. It proves the equivalence between stationary points of the optimization problem and equilibria of PI--PGD, and provides a contraction-based convergence analysis for affine equality constraints, yielding linear-exponential convergence to the equilibrium. The authors validate the approach on equality-constrained LASSO and nonlinear equality constraints, demonstrating robust performance and practical viability beyond strict assumptions. This work connects optimization with control-theoretic feedback design, offering a principled framework for solving constrained, possibly non-smooth problems with provable convergence properties and broad applicability in engineering and machine learning contexts.

Abstract

This paper studies equality-constrained composite minimization problems. This class of problems, capturing regularization terms and inequality constraints, naturally arises in a wide range of engineering and machine learning applications. To tackle these optimization problems, inspired by recent results, we introduce the \emph{proportional--integral proximal gradient dynamics} (PI--PGD): a closed-loop system where the Lagrange multipliers are control inputs and states are the problem decision variables. First, we establish the equivalence between the stationary points of the minimization problem and the equilibria of the PI--PGD. Then for the case of affine constraints, by leveraging tools from contraction theory we give a comprehensive convergence analysis for the dynamics, showing linear--exponential convergence towards the equilibrium. That is, the distance between each solution and the equilibrium is upper bounded by a function that first decreases linearly and then exponentially. Our findings are illustrated numerically on a set of representative examples, which include an exploratory application to nonlinear equality constraints.

Paper Structure

This paper contains 15 sections, 7 theorems, 47 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Mathematical Preliminaries
  3. Norms, Logarithmic Norms and Weak Pairings.
  4. Convex Analysis and Proximal Operators.
  5. Contraction Theory for Dynamical Systems.
  6. Equality-Constrained Composite Optimization via Feedback Control
  7. Convergence of the PI Proximal-Gradient Dynamics
  8. (i) $\mathbf{Q_{11} \succ 0}$.
  9. (ii) $\mathbf{Q_{22} - Q_{12}^\top Q_{11}^{-1}Q_{12} \succeq 0}$.
  10. Numerical Examples and Applications
  11. Equality-Constrained LASSO
  12. Nonlinear-Equality-Constrained LASSO
  13. Discussions and Conclusions
  14. Instrumental Result
  15. An Application to Entropic Regularized Optimal Transport

Key Result

Theorem 1

Consider problem eq:eq_constrained_non_smooth and let $x^{\star} \in \mathbb{R}^{n}$ be a local minimum satisfying $h(x^{\star}) = \hbox{\fontencoding{U}0}_m$. Assume that $x^{\star}$ is regular, that is, the rows of $Dh(x^\star)$ are linearly independent. Then, there exists a unique $\lambda^{\star

Figures (10)

  • Figure 1: Closed-loop system for equality-constrained OPs: $\dot{x} = F(x, \lambda)$ has the stationary points of the Lagrangian as equilibria. The feedback controller, $C$, is designed to drive $y$ toward the reference value $r = 0$, so that $h(x) = \hbox{\fontencoding{U}0}_m$.
  • Figure 2: PI--PGD: Closed-loop dynamics composed by system \ref{['eq:system_prox_FB_controlled']} and the PI controller \ref{['eq:pi_controller']}.
  • Figure 3: Trajectories of the dynamics \ref{['eq:pipgd_constrained_lasso']} solving the constrained minimization problem \ref{['eq:constrained_lasso']}. The figure shows the trajectories of the primal variables $x(t)$ (top) and two dual variables $\lambda(t)$ (middle), starting from $z^1_0$ and $z^2_0$ as solid and dashed curves, respectively. The cvxpy optimal values are shown as dots. The bottom panel displays the constraint residual $A x(t) - b$ over time.
  • Figure 4: Mean and standard deviation of $\log(\|z(t) - z^\star\|_{P})$ across 150 simulations. Consistent with Theorem \ref{['thm:GLin-ExpS']}, convergence is linearly-exponentially bounded.
  • Figure 5: Trajectories of \ref{['eq:pi-pgd_non_lin2']} solving LASSO with nonlinear equality constraints \ref{['eq:opt_problem_non_lin2']}. The panel shows the trajectories of the primal variables $x(t)$ (top) and two dual variables $\lambda(t)$ (middle), starting from random initial conditions. The SLSQP optimal values are shown as cross. The bottom panel displays the constraint $h(x)$ over time. The trajectories effectively converge to $z^\star$, and constraints are satisfied after a short settling time.
  • ...and 5 more figures

Theorems & Definitions (24)

  • Definition 1: Proximal Operator
  • Theorem 1: First-order necessary conditions
  • Remark 1: Stationary points vs. minimizers
  • Definition 2: Contracting dynamics
  • Theorem 2: Linear-exponential convergence of globally-weakly and locally-strongly contracting dynamics
  • Remark 2
  • Definition 3: Stationary point of \ref{['eq:eq_constrained_non_smooth']}
  • Remark 3: First-order necessary and sufficient conditions for convex OP
  • Remark 4
  • Lemma 1: Linking the stationary point of \ref{['eq:eq_constrained_non_smooth']} and the equilibria of \ref{['eq:system_prox_FB_controlled']}
  • ...and 14 more