Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Rank GEMM: Efficient Matrix Multiplication via Low-Rank Approximation with FP8 Acceleration

Alfredo Metere

TL;DR

Low-Rank GEMM introduces a production-ready framework that uses low-rank approximations to reduce the complexity and memory footprint of large-scale GEMMs while exploiting FP8 hardware acceleration. The method adaptively selects ranks and decomposition methods, and leverages TensorCores with FP8 storage and FP32 accumulation to balance throughput and numerical stability. Across benchmarks on an RTX 4090, it achieves up to 378 TFLOPS and 75% memory savings, with a crossover to the LowRank approach around matrices of size $N \approx 10^4$ where bandwidth becomes the primary bottleneck. The work demonstrates that near-bandwidth-limited performance is achievable for large-scale matrix multiplications, enabling training and deployment of larger models and more throughput-efficient inference in real-world ML workloads.

Abstract

Large matrix multiplication is a cornerstone of modern machine learning workloads, yet traditional approaches suffer from cubic computational complexity (e.g., $\mathcal{O}(n^3)$ for a matrix of size $n\times n$). We present Low-Rank GEMM, a novel approach that leverages low-rank matrix approximations to achieve sub-quadratic complexity while maintaining hardware-accelerated performance through FP8 precision and intelligent kernel selection. On a NVIDIA RTX 4090, our implementation achieves up to 378 TFLOPS on matrices up to $N=20480$, providing 75\% memory savings and $7.8\times$ speedup over PyTorch FP32 for large matrices. The system automatically adapts to hardware capabilities, selecting optimal decomposition methods (SVD, randomized SVD) and precision levels based on matrix characteristics and available accelerators. Comprehensive benchmarking on NVIDIA RTX 4090 demonstrates that Low-Rank GEMM becomes the fastest approach for matrices $N\geq10240$, surpassing traditional cuBLAS implementations through memory bandwidth optimization rather than computational shortcuts.

Low-Rank GEMM: Efficient Matrix Multiplication via Low-Rank Approximation with FP8 Acceleration

TL;DR

Low-Rank GEMM introduces a production-ready framework that uses low-rank approximations to reduce the complexity and memory footprint of large-scale GEMMs while exploiting FP8 hardware acceleration. The method adaptively selects ranks and decomposition methods, and leverages TensorCores with FP8 storage and FP32 accumulation to balance throughput and numerical stability. Across benchmarks on an RTX 4090, it achieves up to 378 TFLOPS and 75% memory savings, with a crossover to the LowRank approach around matrices of size where bandwidth becomes the primary bottleneck. The work demonstrates that near-bandwidth-limited performance is achievable for large-scale matrix multiplications, enabling training and deployment of larger models and more throughput-efficient inference in real-world ML workloads.

Abstract

Large matrix multiplication is a cornerstone of modern machine learning workloads, yet traditional approaches suffer from cubic computational complexity (e.g., for a matrix of size ). We present Low-Rank GEMM, a novel approach that leverages low-rank matrix approximations to achieve sub-quadratic complexity while maintaining hardware-accelerated performance through FP8 precision and intelligent kernel selection. On a NVIDIA RTX 4090, our implementation achieves up to 378 TFLOPS on matrices up to , providing 75\% memory savings and speedup over PyTorch FP32 for large matrices. The system automatically adapts to hardware capabilities, selecting optimal decomposition methods (SVD, randomized SVD) and precision levels based on matrix characteristics and available accelerators. Comprehensive benchmarking on NVIDIA RTX 4090 demonstrates that Low-Rank GEMM becomes the fastest approach for matrices , surpassing traditional cuBLAS implementations through memory bandwidth optimization rather than computational shortcuts.

Paper Structure

This paper contains 51 sections, 10 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Low-Rank Matrix Approximation
  4. Hardware-Accelerated Matrix Multiplication
  5. Approximate Computing in ML
  6. Methodology
  7. Low-Rank Matrix Approximation
  8. Adaptive Rank Selection
  9. Hardware Acceleration
  10. FP8 Precision Support
  11. Importance of FP32 Accumulation
  12. TensorCore Optimization
  13. Role of FP16 in FP8 Kernels
  14. Implementation Architecture
  15. Experimental Setup
  16. ...and 36 more sections

Figures (1)

  • Figure 1: RTX 4090 Large Scale Performance: Time-to-solution, throughput, error, and speedup vs matrix size ($\log_2$ scale). LowRank Auto achieves up to 378 TFLOPS at N=20480, becoming the fastest method for $N\geq10240$.