Optimization on product manifolds under a preconditioned metric

TL;DR

This work develops a general framework for optimization on product manifolds using preconditioned metrics that approximate second-order information. It introduces three design families—exact block diagonal, left-right, and Gauss–Newton type preconditioning—and tailors them to CCA, truncated SVD, and tensor ring completion, with theoretical guarantees on Hessian conditioning and accelerated local convergence. By deriving explicit gradients, Hessians, and projection operators under these metrics, the authors demonstrate substantial reductions in Hessian condition numbers and faster convergence in practice, while maintaining comparable per-iteration costs. The results suggest that meticulous metric design is a practical lever to improve Riemannian optimization performance across a spectrum of structured problems.

Abstract

Since optimization on Riemannian manifolds relies on the chosen metric, it is appealing to know that how the performance of a Riemannian optimization method varies with different metrics and how to exquisitely construct a metric such that a method can be accelerated. To this end, we propose a general framework for optimization problems on product manifolds endowed with a preconditioned metric, and we develop Riemannian methods under this metric. Generally, the metric is constructed by an operator that aims to approximate the diagonal blocks of the Riemannian Hessian of the cost function. We propose three specific approaches to design the operator: exact block diagonal preconditioning, left and right preconditioning, and Gauss--Newton type preconditioning. Specifically, we tailor new preconditioned metrics and adapt the proposed Riemannian methods to the canonical correlation analysis and the truncated singular value decomposition problems, which provably accelerate the Riemannian methods. Additionally, we adopt the Gauss--Newton type preconditioning to solve the tensor ring completion problem. Numerical results among these applications verify that a delicate metric does accelerate the Riemannian optimization methods.

Figures (8)

• Figure 1: Left: sequences generated by the Riemannian gradient descent method under two metrics for $\mathbf{B}=\mathop{\mathrm{diag}}\nolimits(2^2,3^2,1)$ and $\mathbf{b}=(1,1,1)$. Right: the condition number of $\mathop{\mathrm{Hess}}\nolimits_{g_{\lambda}}\!f(\mathbf{x}^*)$ for $\lambda\in(-1/8,1]$. Blue marker: the Euclidean metric; green marker: the scaled metric.
• Figure 1: Numerical results for CCA problem for $d_x=800$, $d_y=400$, and $m=5$. Left: RGD. Right: RCG. Each method is tested for 10 runs.
• Figure 1: Numerical results for the SVD problem for $m=1000$, $n=500$, and $p=10$. Left: RGD. Right: RCG. Each method is tested for 10 runs
• Figure 1: Training and test errors for TR rank $\mathbf{r}^*=(5,5,5)$. Each method is tested for 10 runs
• Figure 2: A new metric on the product manifold $\mathcal{M}$.

Theorems & Definitions (23)

• Example 1.1
• Definition 2.1: critical points
• Proposition 2.2
• Proof 1
• Definition 2.3: second-order critical points
• Definition 2.4: Armijo backtracking line search
• Theorem 2.5
• Proposition 3.1
• Proposition 3.2
• Remark 3.3