Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HadaCore: Tensor Core Accelerated Hadamard Transform Kernel

Krish Agarwal, Rishi Astra, Adnan Hoque, Mudhakar Srivatsa, Raghu Ganti, Less Wright, Sijia Chen

TL;DR

HadaCore presents a Tensor Core-accelerated FWHT kernel that preserves the asymptotic $O(mn\log(n))$ complexity while exploiting GPU hardware to reduce data movement and synchronization bottlenecks. By employing a 16-element Hadamard base case and transposition-based data reshaping, the method achieves substantial speedups on Nvidia A100 and H100 GPUs, with up to $3.5\times$ and $3.6\times$ improvements over a state-of-the-art library. It also demonstrates comparable numerical accuracy for FP16/BF16 and effective end-to-end performance in FP8-attention LLMS like Llama-3.1, validating the practical utility for quantized transformer workloads. The work lays groundwork for future integration with Triton and further Hopper-era optimizations, potentially enabling even richer fused kernels for Hadamard transforms and quantization pipelines.

Abstract

We present HadaCore, a modified Fast Walsh-Hadamard Transform (FWHT) algorithm optimized for the Tensor Cores present in modern GPU hardware. HadaCore follows the recursive structure of the original FWHT algorithm, achieving the same asymptotic runtime complexity but leveraging a hardware-aware work decomposition that benefits from Tensor Core acceleration. This reduces bottlenecks from compute and data exchange. On Nvidia A100 and H100 GPUs, HadaCore achieves speedups of 1.1-1.4x and 1.0-1.3x, with a peak gain of 3.5x and 3.6x respectively, when compared to the existing state-of-the-art implementation of the original algorithm. We also show that when using FP16 or BF16, our implementation is numerically accurate, enabling comparable accuracy on MMLU benchmarks when used in an end-to-end Llama3 inference run with quantized (FP8) attention.

HadaCore: Tensor Core Accelerated Hadamard Transform Kernel

TL;DR

HadaCore presents a Tensor Core-accelerated FWHT kernel that preserves the asymptotic complexity while exploiting GPU hardware to reduce data movement and synchronization bottlenecks. By employing a 16-element Hadamard base case and transposition-based data reshaping, the method achieves substantial speedups on Nvidia A100 and H100 GPUs, with up to and improvements over a state-of-the-art library. It also demonstrates comparable numerical accuracy for FP16/BF16 and effective end-to-end performance in FP8-attention LLMS like Llama-3.1, validating the practical utility for quantized transformer workloads. The work lays groundwork for future integration with Triton and further Hopper-era optimizations, potentially enabling even richer fused kernels for Hadamard transforms and quantization pipelines.

Abstract

We present HadaCore, a modified Fast Walsh-Hadamard Transform (FWHT) algorithm optimized for the Tensor Cores present in modern GPU hardware. HadaCore follows the recursive structure of the original FWHT algorithm, achieving the same asymptotic runtime complexity but leveraging a hardware-aware work decomposition that benefits from Tensor Core acceleration. This reduces bottlenecks from compute and data exchange. On Nvidia A100 and H100 GPUs, HadaCore achieves speedups of 1.1-1.4x and 1.0-1.3x, with a peak gain of 3.5x and 3.6x respectively, when compared to the existing state-of-the-art implementation of the original algorithm. We also show that when using FP16 or BF16, our implementation is numerically accurate, enabling comparable accuracy on MMLU benchmarks when used in an end-to-end Llama3 inference run with quantized (FP8) attention.

Paper Structure

This paper contains 20 sections, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Hadamard Matrices
  4. Fast Walsh-Hadamard Transform Algorithm
  5. Parallelization and Running on the GPU
  6. fast-hadamard-transform Library
  7. Method
  8. Hadamard Sizes up to 256
  9. Supporting Sizes Larger than 256
  10. Non-Power of 16 Sizes
  11. Algorithm Overview
  12. Results
  13. Performance Tests
  14. End-To-End Tests
  15. Conclusion
  16. ...and 5 more sections

Figures (11)

  • Figure 1: Transformer block showing online (red) and offline rotations (blue) in QuaRot.
  • Figure 2: $1\times 256$ vectors (rows) visualized for rotating by a size-256 Hadamard. The batches (number of rows) can be split among warps of the GPU.
  • Figure 3: The $1\times 256$ vectors now transposed, allowing us to operate on the factor $\frac{n}{256}$.
  • Figure 4: Graphs of runtime and speedup of HadaCore against the Dao AI Lab kernel, measured on an A100-PCIe. Each series represents a different rotation size, and the x-axis is the element count of the input tensor.
  • Figure 5: Graphs of runtime and speedup of HadaCore against the Dao AI Lab kernel, measured on an H100-PCIe. Each series represents a different rotation size, and the x-axis is the element count of the input tensor.
  • ...and 6 more figures