Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variable reduction as a nonlinear preconditioning approach for optimization problems

Gabriele Ciaremalla, Tommaso Vanzan

TL;DR

It is argued that nonlinear elimination could be used to reduce the number of optimization variables by artificially constraining them to satisfy a subset of the optimality conditions, and a reduced objective function is derived which can now be minimized with any optimization algorithm.

Abstract

When considering an unconstrained minimization problem, a standard approach is to solve the optimality system with a Newton method possibly preconditioned by, e.g., nonlinear elimination. In this contribution, we argue that nonlinear elimination could be used to reduce the number of optimization variables by artificially constraining them to satisfy a subset of the optimality conditions. Consequently, a reduced objective function is derived which can now be minimized with any optimization algorithm. By choosing suitable variables to eliminate, the conditioning of the reduced optimization problem is largely improved. We here focus in particular on a right preconditioned gradient descent and show theoretical and numerical results supporting the validity of the presented approach.

Variable reduction as a nonlinear preconditioning approach for optimization problems

TL;DR

It is argued that nonlinear elimination could be used to reduce the number of optimization variables by artificially constraining them to satisfy a subset of the optimality conditions, and a reduced objective function is derived which can now be minimized with any optimization algorithm.

Abstract

When considering an unconstrained minimization problem, a standard approach is to solve the optimality system with a Newton method possibly preconditioned by, e.g., nonlinear elimination. In this contribution, we argue that nonlinear elimination could be used to reduce the number of optimization variables by artificially constraining them to satisfy a subset of the optimality conditions. Consequently, a reduced objective function is derived which can now be minimized with any optimization algorithm. By choosing suitable variables to eliminate, the conditioning of the reduced optimization problem is largely improved. We here focus in particular on a right preconditioned gradient descent and show theoretical and numerical results supporting the validity of the presented approach.
Paper Structure (4 sections, 1 theorem, 16 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 4 sections, 1 theorem, 16 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Nonlinear elimination and a right preconditioned gradient descent
  3. Inexact variable reduction
  4. Conclusions

Key Result

proposition 1

For an unconstrained quadratic optimization problem, the right PGD method always has a better asymptotic convergence rate than the standard GD method.

Figures (1)

  • Figure 1: Comparison between the convergence of GD and PGD for a quadratic problem. $\kappa_2(A)= 1.001\cdot 10^3$, $\kappa_2(S)=10$ (left), $\kappa_2(S)= 6.08\cdot 10^2$ (center). Right panel convergence of GD and PGD for the strongly convex optimization problem \ref{['eq:strongly_convex']} with $n=100$ and Armijo backtracking.

Theorems & Definitions (1)

  • proposition 1