Mixed Precision Orthogonalization-Free Projection Methods for Eigenvalue and Singular Value Problems

TL;DR

This work introduces Orthogonalization-Free Rayleigh-Ritz (OFRR), a projection framework that computes select eigenvalues and singular values using non-orthogonal bases in mixed precision. By transforming the projection onto a generalized eigenproblem with a mass matrix, OFRR eliminates the need for orthogonalization while maintaining accuracy, aided by Hessenberg-style, inner-product-free basis construction. Comprehensive experiments on kernel and sparse matrices, plus GPU-accelerated implementations, demonstrate that OFRR—especially the Krylov-Hessenberg variant—delivers comparable or superior accuracy to traditional RR in low precision and offers significant speedups. The findings highlight OFRR as a scalable, practical approach for large-scale spectral problems in mixed-precision environments, with clear implications for high-performance computing workflows.

Abstract

Mixed-precision arithmetic offers significant computational advantages for large-scale matrix computation tasks, yet preserving accuracy and stability in eigenvalue problems and the singular value decomposition (SVD) remains challenging. This paper introduces an approach that eliminates orthogonalization requirements in traditional Rayleigh-Ritz projection methods. The proposed method employs non-orthogonal bases computed at reduced precision, resulting in bases computed without inner-products. A primary focus is on maintaining the linear independence of the basis vectors. Through extensive evaluation with both synthetic test cases and real-world applications, we demonstrate that the proposed approach achieves the desired accuracy while fully taking advantage of mixed-precision arithmetic.

Figures (7)

• Figure 1: Relative error plot of subspace iteration with Rayleigh-Ritz projection under different precision options. The test matrix is a Gaussian kernel matrix of size $1000\times 1000$. A concise naming convention is used to denote different options: [MatVec Precision]-[QR Precision].
• Figure 1: The MGS sweep used within Arnoldi (left) and a "right‑looking" variant of MGS for subspace iteration (right). In Arnoldi each processed basis vector updates the single current column via repeated DOT & AXPY operations. In the 'righ-looking' version, once the current column is orthogonal, a single GEMV & GER applies its correction to the entire trailing block at once.
• Figure 1: Relative approximation accuracy using different algorithms with different precisions and true leading eigenvalues. The test matrices are Gaussian kernel matrices of size $1000\times1000$ with $f=0.2$, $l=10$, $s=0.01$ (test configuration 1) and $f=0.2$, $l = 100$, $s = 0$ (test configuration 2).
• Figure 2: Condition number for bases computed by four different methods: no stabilization (-X), MGS with re-orthogonalization, CGS with re-orthogonalization, and Hessenberg. Tests are performed on multiple kernel matrices, each sized $1000\times 1000$, with length scales varying from $1$ to $100$.
• Figure 2: Relative residual norm using different algorithms with different precisions. The test matrices are from SuiteSparse matrix collection.

Theorems & Definitions (2)

• Theorem 1
• Proof 1