Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Convergence of Constrained Gradient Method

Danqing Zhou, Hongmei Chen, Shiqian Ma, Junfeng Yang

TL;DR

This paper strengthens convergence guarantees for the constrained gradient method (CGM) under weaker, more practical assumptions for both strongly convex optimization and strongly monotone variational inequality problems with general functional constraints. It removes reliance on infeasible conditions, such as bounds on gradient norms or auxiliary norm constraints, and shows that CGM achieves $\mathcal{O}(1/T)$-type rates (and their tilde variants) for objective/optimality and $\tilde{\mathcal{O}}(1/T)$ feasibility under constant and diminishing step sizes. The analysis covers both minimization and VI settings with relaxed Lipschitz conditions on $F$ and non-Lipschitz constraints, while ensuring boundedness of iterates without requiring a bounded feasible set. Numerical experiments on a resource allocation problem and a high-dimensional bilinear game corroborate the theoretical results and demonstrate competitive performance against projection-based methods, highlighting CGM’s practicality for large-scale constrained optimization and VI problems.

Abstract

The constrained gradient method (CGM) has recently been proposed to solve convex optimization and monotone variational inequality (VI) problems with general functional constraints. While existing literature has established convergence results for CGM, the assumptions employed therein are quite restrictive; in some cases, certain assumptions are mutually inconsistent, leading to gaps in the underlying analysis. This paper aims to derive rigorous and improved convergence guarantees for CGM under weaker and more reasonable assumptions, specifically in the context of strongly convex optimization and strongly monotone VI problems. Preliminary numerical experiments are provided to verify the validity of CGM and demonstrate its efficacy in addressing such problems.

On the Convergence of Constrained Gradient Method

TL;DR

This paper strengthens convergence guarantees for the constrained gradient method (CGM) under weaker, more practical assumptions for both strongly convex optimization and strongly monotone variational inequality problems with general functional constraints. It removes reliance on infeasible conditions, such as bounds on gradient norms or auxiliary norm constraints, and shows that CGM achieves -type rates (and their tilde variants) for objective/optimality and feasibility under constant and diminishing step sizes. The analysis covers both minimization and VI settings with relaxed Lipschitz conditions on and non-Lipschitz constraints, while ensuring boundedness of iterates without requiring a bounded feasible set. Numerical experiments on a resource allocation problem and a high-dimensional bilinear game corroborate the theoretical results and demonstrate competitive performance against projection-based methods, highlighting CGM’s practicality for large-scale constrained optimization and VI problems.

Abstract

The constrained gradient method (CGM) has recently been proposed to solve convex optimization and monotone variational inequality (VI) problems with general functional constraints. While existing literature has established convergence results for CGM, the assumptions employed therein are quite restrictive; in some cases, certain assumptions are mutually inconsistent, leading to gaps in the underlying analysis. This paper aims to derive rigorous and improved convergence guarantees for CGM under weaker and more reasonable assumptions, specifically in the context of strongly convex optimization and strongly monotone VI problems. Preliminary numerical experiments are provided to verify the validity of CGM and demonstrate its efficacy in addressing such problems.

Paper Structure

This paper contains 18 sections, 7 theorems, 49 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Results adapted from MJ22.
  3. Results adapted from ZHM25.
  4. Motivation and contributions.
  5. Organization.
  6. Preliminaries
  7. Notation.
  8. Optimality measures.
  9. Useful lemmas.
  10. CGM for Strongly Convex Optimization
  11. CGM for Strongly Monotone VIs
  12. Numerical Experiments
  13. Resource Allocation Problem
  14. High-Dimensional Bilinear Game
  15. Conclusions
  16. ...and 3 more sections

Key Result

Lemma 2.1

For all $x\in \mathcal{C}$ and $\alpha>0$, we have $\alpha(x-x^{t})\in V_{\alpha}(x^t)$.

Figures (4)

  • Figure 1: Convergence results of CGM for the RAP with varying iteration counts: small $T=\{100,150,200,250\}$ and large $T=\{1500,2000,2500,3000\}$. Left: absolute function value residual; Right: constraint violation with $m=d+4$.
  • Figure 2: Convergence results of CGM for the RAP: constant step size $\eta_t=\eta=\log T/(\mu T)$ (with $T=2000$) versus diminishing step size $\eta_t = 1/\mu(t+\kappa)$. Left: absolute function value residual; Middle: function value residual; Right: constraint violation with $m=d+4$.
  • Figure 3: Convergence results of CGM for the HBG problem with different $\beta$ values. Left: optimality residual; Right: constraint violation with $m=2d+5$ (including the auxiliary constraint).
  • Figure 4: Comparison results of CGM versus GDA and EG methods on the HBG problem: relative error in distance to the optimum under fixed iteration counts and fixed CPU time constraints. Left: relative error v.s. iteration; Right: relative error v.s. CPU time.

Theorems & Definitions (18)

  • Definition 2.1: Weak $\epsilon$-solution
  • Lemma 2.1
  • proof
  • Lemma 2.2
  • proof
  • Remark 3.1
  • Remark 3.2
  • Lemma 3.1: Bounding $\|v^t\|^2$ via function values
  • proof
  • Lemma 3.2: Boundedness of $\|v^t\|$ and $\|x^t-x^*\|$
  • ...and 8 more