$XX^{t}$ Can Be Faster

TL;DR

This work targets the efficient computation of $XX^{t}$ for $X\in\mathbb{R}^{n\times m}$ by introducing RXTX, a recursive $4\times 4$-block algorithm that reduces multiplications and total operations by about 5% versus the state of the art. RXTX's costs are analyzed via recurrences $R(n)=8R(n/4)+26M(n/4)$ and $R(n)\sim\frac{26}{41}M(n)+\frac{15}{41}n^{3/2}$, yielding an asymptotic constant of $\frac{26}{41}\approx0.6341$, a modest improvement over Strassen-inspired constants. The method also achieves non-asymptotic gains (e.g., $R(4)=34$ vs $R(4)=38$) and is supported by empirical runtime gains (e.g., average $2.524$ s vs $2.778$ s for large dense matrices) with high reliability. RXTX was discovered by RL-guided Large Neighborhood Search combined with a MILP-based pipeline, demonstrating how structured search can uncover efficient algorithms for structured matrix products like $XX^{t}$ with practical impact on covariance/Gram computations and potential applications in ML training pipelines.

Abstract

We present RXTX, a new algorithm for computing the product of matrix by its transpose $XX^{t}$ for $X\in \mathbb{R}^{n\times m}$. RXTX uses $5\%$ fewer multiplications and $5\%$ fewer operations (additions and multiplications) than State-of-the-Art algorithms. Note that the accelerations not only holds asymptotically for large matrices with $n \rightarrow \infty$, but also for small matrices including $n = 4$. The algorithm was discovered by combining Machine Learning-based search methods with Combinatorial Optimization.

Figures (5)

• Figure 1: Comparison of number of multiplications of RXTX to previous SotA and naive algorithm.
• Figure 2: Comparison of number of multiplications of RXTX with optimal cutoff to previous SotA and naive algorithm.
• Figure 3: Comparison of number of operations of RXTX to recursive Strassen and naive algorithm. RXTX outperforms recursive Strassen for $n \geq 256$ and naive algorithm for $n \geq 1024$.
• Figure 4: Comparison of algorithms with optimal cutoffs i.e. for small enough matrices in recursion switch to the algorithm with least operations. RXTX outperforms naive algorithm for $n \geq 32$ and SotA for $n \geq 256$.
• Figure 5: The average runtime for RXTX is $2.524$s, which is 9% faster than average runtime of specific BLAS routine $2.778$s. RXTX was faster in $99\%$ of the runs.

Theorems & Definitions (4)

• Theorem 1
• proof
• Theorem 2
• proof