Inertial Quasi-Newton Methods for Monotone Inclusion: Efficient Resolvent Calculus and Primal-Dual Methods

TL;DR

A new efficient resolvent calculus for a low-rank perturbed standard metric, which accounts exactly for quasi-Newton metrics is developed, which proves the convergence of the algorithms, including linear convergence rates in case one of the two considered operators is strongly monotone.

Abstract

We introduce an inertial quasi-Newton Forward-Backward Splitting Algorithm to solve a class of monotone inclusion problems. While the inertial step is computationally cheap, in general, the bottleneck is the evaluation of the resolvent operator. A change of the metric makes its computation hard even for (otherwise in the standard metric) simple operators. In order to fully exploit the advantage of adapting the metric, we develop a new efficient resolvent calculus for a low-rank perturbed standard metric, which accounts exactly for quasi-Newton metrics. Moreover, we prove the convergence of our algorithms, including linear convergence rates in case one of the two considered operators is strongly monotone. Beyond the general monotone inclusion setup, we instantiate a novel inertial quasi-Newton Primal-Dual Hybrid Gradient Method for solving saddle point problems. The favourable performance of our inertial quasi-Newton PDHG method is demonstrated on several numerical experiments in image processing.

Figures (3)

• Figure 1: We compare convergence of the inertial quasi-Newton PDHG (IQN-FBS) to other algorithms in the Table \ref{['tab:my-table']} with $\mu = 0.0001$, $\tau =0.05$, $\sigma =0 .05$ and inertial parameter $\alpha_0 =10$, $\alpha_k = \frac{10}{k^{1.1}(\max\{\Vert z_k-z_{k-1} \Vert_{},\Vert z_k-z_{k-1} \Vert_{}^2\})}$. The left plot depicts the convergence against the number of iterations, while the right plot shows the convergence with respect to time (seconds). We observe that our two quasi-Newton type algorithm QN-, and IQN-FBS clearly outperform the original FBS and IFBS algorithm.
• Figure 2: We compare convergence of the inertial quasi-Newton PDHG (IQN-FBS) to other algorithms in the Table \ref{['tab:my-table']} with $\mu = 0.5$, $\tau =0.01I$, $\sigma =0.01$ and the extrapolation parameter $\alpha_0 =1$, $\alpha_k =\min\{ \frac{10}{k^{1.1}(\max\{\Vert z_k-z_{k-1} \Vert_{},\Vert z_k-z_{k-1} \Vert_{}^2\})},1\}$. Our quasi-Newton type algorithm IQN-FBS can converge faster than the original FBS and IFBS algorithm.
• Figure 3: We compare convergence of the inertial quasi-Newton PDHG (IQN-FBS) to other algorithms in the Table \ref{['tab:my-table']} with $\mu = 0.1$, $\tau =0.1$, $\sigma =0.1$ and extrapolation parameter $\alpha_0 =10$, $\alpha_k = \frac{10}{\max\{k^{1.1},k^{1.1}\Vert z_k-z_{k-1} \Vert_{}^{2}\}}$. The plot shows that all algorithms converge linearly and faster than $O(\frac{1}{1.1^k})$.

Theorems & Definitions (57)

• Proposition 2.1
• Proposition 2.2
• Lemma 2.3
• proof
• Lemma 2.4
• proof
• Proposition 2.5: Variable Metric quasi-Fejér monotone sequence combettes2012variable
• Lemma 2.6: A duality result for operators Attouch
• Lemma 2.7
• Theorem 3.1