Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ozaki Scheme II: A GEMM-oriented emulation of floating-point matrix multiplication using an integer modular technique

Katsuhisa Ozaki, Yuki Uchino, Toshiyuki Imamura

TL;DR

Ozaki scheme II introduces a CRT-based, GEMM-oriented framework for high-precision matrix multiplication emulation. By reconstructing A'B' from modular products with carefully chosen coprime moduli, it dramatically reduces the number of GEMM calls while allowing explicit precision control. Empirical results on GPUs with INT8 Tensor Cores and CPUs with FP64 indicate competitive or superior performance to native FP64 and the conventional Ozaki scheme, with notable throughput gains and scalable multi-word arithmetic support. This approach offers practical benefits for high-precision linear algebra on heterogeneous hardware, particularly when leveraging fast low-precision matrix engines.

Abstract

This paper addresses emulation algorithms for matrix multiplication. General Matrix-Matrix Multiplication (GEMM), a fundamental operation in the Basic Linear Algebra Subprograms (BLAS), is typically optimized for specific hardware architectures. The Ozaki scheme is a well-established GEMM-based emulation method for matrix multiplication, wherein input matrices are decomposed into several low-precision components to ensure that the resulting matrix product is computed exactly through numerical operations. This study proposes a novel GEMM-based emulation method for matrix multiplication that leverages the Chinese Remainder Theorem. The proposed method inherits the computational efficiency of highly optimized GEMM routines and further enables control over the number of matrix multiplications, which can enhance computational accuracy. We present numerical experiments featuring INT8 Tensor Core operations on GPUs and FP64 arithmetic on CPUs as case studies. The results demonstrate that FP64 emulation using the proposed method achieves performance levels of up to 7.4 to 9.8 TFLOPS on the NVIDIA RTX 4090 and 56.6 to 80.2 TFLOPS on the NVIDIA GH200, exceeding the measured performance of native FP64 arithmetic. Furthermore, for FP64 computations on CPUs, the proposed method achieved up to a 2.3x speedup in emulating quadruple-precision arithmetic compared to the conventional Ozaki scheme.

Ozaki Scheme II: A GEMM-oriented emulation of floating-point matrix multiplication using an integer modular technique

TL;DR

Ozaki scheme II introduces a CRT-based, GEMM-oriented framework for high-precision matrix multiplication emulation. By reconstructing A'B' from modular products with carefully chosen coprime moduli, it dramatically reduces the number of GEMM calls while allowing explicit precision control. Empirical results on GPUs with INT8 Tensor Cores and CPUs with FP64 indicate competitive or superior performance to native FP64 and the conventional Ozaki scheme, with notable throughput gains and scalable multi-word arithmetic support. This approach offers practical benefits for high-precision linear algebra on heterogeneous hardware, particularly when leveraging fast low-precision matrix engines.

Abstract

This paper addresses emulation algorithms for matrix multiplication. General Matrix-Matrix Multiplication (GEMM), a fundamental operation in the Basic Linear Algebra Subprograms (BLAS), is typically optimized for specific hardware architectures. The Ozaki scheme is a well-established GEMM-based emulation method for matrix multiplication, wherein input matrices are decomposed into several low-precision components to ensure that the resulting matrix product is computed exactly through numerical operations. This study proposes a novel GEMM-based emulation method for matrix multiplication that leverages the Chinese Remainder Theorem. The proposed method inherits the computational efficiency of highly optimized GEMM routines and further enables control over the number of matrix multiplications, which can enhance computational accuracy. We present numerical experiments featuring INT8 Tensor Core operations on GPUs and FP64 arithmetic on CPUs as case studies. The results demonstrate that FP64 emulation using the proposed method achieves performance levels of up to 7.4 to 9.8 TFLOPS on the NVIDIA RTX 4090 and 56.6 to 80.2 TFLOPS on the NVIDIA GH200, exceeding the measured performance of native FP64 arithmetic. Furthermore, for FP64 computations on CPUs, the proposed method achieved up to a 2.3x speedup in emulating quadruple-precision arithmetic compared to the conventional Ozaki scheme.

Paper Structure

This paper contains 12 sections, 48 equations, 10 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation and Previous Study
  3. Notation
  4. Conventional Ozaki scheme
  5. Chinese Remainder Theorem
  6. Proposed Method
  7. Using INT8 TCs for matrix multiplication
  8. Using FP64 for matrix multiplication
  9. Numerical Experiment
  10. Using INT8 TCs for FP64 emulation
  11. Using FP64 for Quad-word arithmetic emulation
  12. Conclusion

Figures (10)

  • Figure 1: Maximum relative error for each slice (left) and the number of matrix multiplications (right)
  • Figure 2: Unique / Non-unique candidate for $A'B'$
  • Figure 3: $k:=k_A = k_B$ in \ref{['eq:easy_k2']} from number of moduli using INT8 Tensor Cores for matrix multiplication
  • Figure 4: $k:=k_A = k_B$ from number of moduli \ref{['eq:easy_k2']} using FP64 for matrix multiplication
  • Figure 5: Accuracy comparison for $p = r = 1024$ and $q \in \{1024, 2048, 4096, 8192, 16384\}$ on NVIDIA GH200 Grace Hopper Superchip
  • ...and 5 more figures