Can Tensor Cores Benefit Memory-Bound Kernels? (No!)
Lingqi Zhang, Jiajun Huang, Sheng Di, Satoshi Matsuoka, Mohamed Wahib
TL;DR
This work addresses whether tensor cores provide meaningful speedups for memory-bound kernels. It combines a theoretical roofline- and machine-balance–based analysis with empirical tests on SCALE, SpMV, and stencil benchmarks across A100 and GH200 GPUs, deriving a FP64 upper bound of $1.33\times$ speedup for memory-bound kernels. The results show tensor cores typically underperform compared with CUDA cores for these workloads, with speedups remaining in the range $[1,1.33)$ and often closer to 1. The study emphasizes prioritizing memory-access optimizations and compute-bound strategies for tensor-core usage, clarifying expectations for tensor cores outside dense linear algebra.
Abstract
Tensor cores are specialized processing units within GPUs that have demonstrated significant efficiency gains in compute-bound applications such as Deep Learning Training by accelerating dense matrix operations. Given their success, researchers have attempted to extend tensor core capabilities beyond dense matrix computations to other computational patterns, including memory-bound kernels. Recent studies have reported that tensor cores can outperform traditional CUDA cores even on memory-bound kernels, where the primary performance bottleneck is not computation. In this research, we challenge these findings through both theoretical and empirical analysis. Our theoretical analysis reveals that tensor cores can achieve a maximum speedup of only 1.33x over CUDA cores for memory-bound kernels in double precision (for V100, A100, and H100 GPUs). We validate this theoretical limit through empirical analysis of three representative memory-bound kernels-STREAM Scale, SpMV, and stencil. We demonstrate that optimizing memory-bound kernels using tensor cores does not yield sound performance improvements over CUDA cores.