$S^{\top\!}S$-SVD via Sketching and the Nearest $S^{\top\!}S$-orthogonal Matrix
Davide Palitta, Valeria Simoncini
TL;DR
This work introduces the $S^{\top}S$-SVD, a probabilistic SVD-like decomposition that uses left factors $W$ that are $S^{\top}S$-orthonormal, with $A=W\Theta V^\top$ and $S$ a subspace embedding. It shows that the nonzero $\theta_i$ satisfy $(1-\varepsilon)^{1/2}\sigma_i \le \theta_i \le (1+\varepsilon)^{1/2}\sigma_i$, ensuring that spectral information is preserved with high probability and enabling low-cost approximations. The decomposition can be computed via the SVD of $SA$ or via a randomized QR factorization followed by an SVD of $R$, and it provides a one-pass option; it also yields a way to evaluate distances from Euclidean orthogonality of sketched orthogonal factors and to bound the error of nearest-sketch orthogonal matrices. The paper applies the $S^{\top}S$-SVD to bound the quality of sketched QR and to solve the nearest-$S^{\top}S$-orthogonal matrix problem, with probabilistic bounds on the solution quality. Overall, the $S^{\top}S$-SVD offers a unified framework to analyze sketching-based least-squares and orthogonalization methods, enabling fast, probabilistic, full-matrix factorizations and aiding rank estimation, truncation, and geometry on the sketched Stiefel manifold.
Abstract
Sketching techniques have gained popularity in numerical linear algebra to accelerate the solution of least squares problems. The so-called $\varepsilon$-subspace embedding property of a sketching matrix $S$ has been largely used to characterize the problem residual norm, since the procedure is no longer optimal in terms of the (classical) Frobenius or Euclidean norm. By building on available results on the SVD of the sketched matrix $SA$ derived by Gilbert, Park, and Wakin (Proc. of SPARS-2013), a novel decomposition of $A$, the $S^{\top\!}S$-SVD, is proposed, which \emph{holds} with high probability, and in which the left singular vectors are orthonormal with respect to a (semi-)norm defined by the sketching matrix $S$. The new decomposition is less expensive to compute than the standard SVD, while preserving the singular values with probabilistic confidence.The $S^{\top\!}S$-SVD appears to be the right tool to analyze the quality of several sketching-based techniques in the literature, for which examples are reported. For instance, it is possible to simply bound the distance from (standard) orthogonality of sketching-based orthogonal matrices in state-of-the-art randomized algorithms for QR factorizations. As an application, the classical problem of the nearest orthogonal matrix is generalized to the new $S^{\top\!}S$-orthogonality, and the $S^{\top\!}S$-SVD is used to solve it. Probabilistic bounds on the quality of the solution are also derived.