Block Coordinate Descent Methods for Optimization under J-Orthogonality Constraints with Applications
Di He, Ganzhao Yuan, Xiao Wang, Pengxiang Xu
TL;DR
This work tackles optimization problems constrained to J-orthogonal matrices, where the constraint takes the form $\mathbf{X}^\top \mathbf{J} \mathbf{X} = \mathbf{J}$ and the objective is a smooth (often finite-sum) function. It introduces JOBCD, a block coordinate descent framework with two variants: GS-JOBCD, which updates 2-by-2 blocks via a Gauss-Seidel-inspired majorization-minimization surrogate, and VR-J-JOBCD, a variance-reduced Jacobi method designed for large-scale finite-sum problems and parallel execution. The authors establish optimality conditions on the J-manifold, global convergence to block-stationary points, and strong convergence under the KL property, with explicit iteration complexities: GS-JOBCD achieves $\mathcal{O}(\Delta_0 N / \varepsilon)$ and VR-J-JOBCD achieves $\mathcal{O}(nN + \Delta_0 \sqrt{N} / \varepsilon)$. Empirically, JOBCD variants outperform state-of-the-art methods across hyperbolic eigenvalue problems, hyperbolic structure probing, and ultrahyperbolic KG embedding, highlighting improved accuracy and efficiency, particularly in large-scale finite-sum settings. The framework thus offers a scalable and theoretically grounded approach to nonconvex optimization under J-orthogonality with broad applicability in hyperbolic and ultrahyperbolic learning tasks.
Abstract
The J-orthogonal matrix, also referred to as the hyperbolic orthogonal matrix, is a class of special orthogonal matrix in hyperbolic space, notable for its advantageous properties. These matrices are integral to optimization under J-orthogonal constraints, which have widespread applications in statistical learning and data science. However, addressing these problems is generally challenging due to their non-convex nature and the computational intensity of the constraints. Currently, algorithms for tackling these challenges are limited. This paper introduces JOBCD, a novel Block Coordinate Descent method designed to address optimizations with J-orthogonality constraints. We explore two specific variants of JOBCD: one based on a Gauss-Seidel strategy (GS-JOBCD), the other on a variance-reduced and Jacobi strategy (VR-J-JOBCD). Notably, leveraging the parallel framework of a Jacobi strategy, VR-J-JOBCD integrates variance reduction techniques to decrease oracle complexity in the minimization of finite-sum functions. For both GS-JOBCD and VR-J-JOBCD, we establish the oracle complexity under mild conditions and strong limit-point convergence results under the Kurdyka-Lojasiewicz inequality. To demonstrate the effectiveness of our method, we conduct experiments on hyperbolic eigenvalue problems, hyperbolic structural probe problems, and the ultrahyperbolic knowledge graph embedding problem. Extensive experiments using both real-world and synthetic data demonstrate that JOBCD consistently outperforms state-of-the-art solutions, by large margins.