Rescaling and unconstrained minimisation of convex quadratic maps

TL;DR

This work studies unconstrained minimisation of convex quadratic maps defined by a SPSD matrix ${\mathbf{Q}}$ and vector ${\boldsymbol{c}}$, by introducing a rescaling-invariant relaxed map ${R}$, ${R(\boldsymbol{x})=\min_{s\ge0}D(s\boldsymbol{x})}$. It establishes that ${R}$ is quasiconvex on ${\mathbb{R}^{N}}$ and pseudoconvex on the cone ${\mathscr{C}}$, and that exact line-search along admissible directions yields a bidimensional optimality property. The authors develop coordinate-descent strategies guided by gradient-based rules in the RKHS associated with ${\mathbf{Q}}$, derive relations between ${R}$ and the original ${D}$ via acceleration factors, and prove asymptotic acceleration bounds under certain conditions. Through extensive experiments on random SPSD maps and structured problems, they demonstrate scenarios where minimising ${R}$ significantly accelerates convergence relative to minimising ${D}$ and even CG, particularly when the asymptotic acceleration term ${\mathfrak{a}_{\infty}}$ is large. The results suggest practical benefits for large-scale quadratic minimisation and highlight directions for further exploration of acceleration phenomena and stochastic variants.

Abstract

We investigate the properties of a class of piecewise-fractional maps arising from the introduction of an invariance under rescaling into convex quadratic maps. The subsequent maps are quasiconvex, and pseudoconvex on specific convex cones; they can be optimised via exact line search along admissible directions, and the iterates then inherit a bidimensional optimality property. We study the minimisation of such relaxed maps via coordinate descents with gradient-based rules, placing a special emphasis on coordinate directions verifying a maximum-alignment property in the reproducing kernel Hilbert spaces related to the underlying positive-semidefinite matrices. In this setting, we illustrate that accounting for the optimal rescaling of the iterates can in certain situations substantially accelerate the unconstrained minimisation of convex quadratic maps.

Figures (9)

• Figure 1: Schematic representation of the situations discussed in Theorem \ref{['thm:RmapOptSS']}. The left plot corresponds to the case $\Upsilon(\boldsymbol{v};\boldsymbol{x})>0$, and the right plot to $\Upsilon(\boldsymbol{v};\boldsymbol{x})\leqslant 0$. In each plot, the grey region indicates the set ${\{\boldsymbol{x}\in\mathbb{R}^{N}|\boldsymbol{c}^{T}\boldsymbol{x}\leqslant 0\}}$, and the grey lines are level sets of the map $R$ on $\mathop{\mathrm{span}}\nolimits\{\boldsymbol{x},\boldsymbol{v}\}$; the vector $\boldsymbol{z}_{\boldsymbol{x},\boldsymbol{v}}$ is characterised in Lemma \ref{['lem:MinRSpan']}.
• Figure 2: Schematic representation of the difference between the iterates of exact line searches for the minimisation of $D$ and $R$. The line searches are implemented from $\boldsymbol{x}$ and along $\boldsymbol{v}$. The iterate obtained for $D$ is $\boldsymbol{u}_{D}$, and $\boldsymbol{u}_{R}$ is the iterate for $R$; the optimally rescaled iterates are also presented. The initial vector $\boldsymbol{x}$ is such that $s_{\boldsymbol{x}}=1$. The grey ellipses are level sets of the map $D$ on $\mathop{\mathrm{span}}\nolimits\{\boldsymbol{x},\boldsymbol{v}\}$.
• Figure 3: For $\mathbf{Q}=\mathbf{X}\mathbf{X}^{T}$, $m=250$, $\mathbf{X} \sim U(2,4)$ and $\boldsymbol{\alpha} \sim U(-2,2)$ (see Example \ref{['ex:Examp1']}), and for the considered minimisation strategies, decay of $D$ as a function of the number of iterations for two different random quadratic maps (top-left and top-middle). For these two maps, and for the minimisation of $R$ via $\mathcal{H}$-CD, the evolution of the acceleration term $\mathcal{A}$ is displayed (bottom-left and bottom-middle). The empirical distributions of the terms $\mathfrak{a}_{\infty}$ and $\mathfrak{a}_{\infty}^{\text{up}}$ for $10{,}000$ randomly-generated quadratic maps are also presented (right).
• Figure 4: Same setting as Figure \ref{['fig:decaywithaccele']}, but with $\boldsymbol{\alpha} \sim U(0,1)$. For a randomly generated quadratic map, decay of $D$ as a function of the number of iterations for the considered minimisation strategies (left). The empirical distributions of $\mathfrak{a}_{\infty}$ and $\mathfrak{a}_{\infty}^{\text{up}}$ for $10{,}000$ random quadratic maps are also presented (right).
• Figure 5: For $\mathbf{Q}=\mathbf{X}\mathbf{X}^{T}+\gamma\mathbf{I}_N$, with $\gamma=0.5$, $5$ and $50$ (left to right), decay of the map $D$ as a function of the number of iterations for the considered minimisation strategies (see Example \ref{['ex:Examp2']}). The matrix $\mathbf{X}$ and the vector $\boldsymbol{c}$ are the same as the ones used in the top-middle plot of Figure \ref{['fig:decaywithaccele']}.

Theorems & Definitions (36)

• Remark 3.1
• Theorem 3.1
• Theorem 3.2
• Lemma 3.1
• proof
• Lemma 3.2
• proof
• proof : Proof of Theorem \ref{['thm:RmapOptSS']}
• Corollary 3.1
• proof