Multi-Strided Access Patterns to Boost Hardware Prefetching

TL;DR

The paper tackles memory-bound compute kernels where hardware prefetching often limits performance. It introduces multi-striding, a method that unrolls loops across non-contiguous axes to create multiple concurrent memory strides, thereby improving prefetcher reach and cache utilization. Through micro-benchmarks and extensive kernel experiments on multiple architectures, the approach yields substantial throughput gains and frequently beats state-of-the-art hand-optimized references like MKL, OpenBLAS, Halide, and OpenCV. The work also provides a practical path for compiler integration by framing multi-striding as an extension of loop transformations and offers open-source tooling for generating multi-strided kernels. The findings suggest significant practical impact for dense linear algebra and stencil computations where memory bandwidth is the bottleneck.

Abstract

Important memory-bound kernels, such as linear algebra, convolutions, and stencils, rely on SIMD instructions as well as optimizations targeting improved vectorized data traversal and data re-use to attain satisfactory performance. On on temporary CPU architectures, the hardware prefetcher is of key importance for efficient utilization of the memory hierarchy. In this paper, we demonstrate that transforming a memory access pattern consisting of a single stride to one that concurrently accesses multiple strides, can boost the utilization of the hardware prefetcher, and in turn improves the performance of memory-bound kernels significantly. Using a set of micro-benchmarks, we establish that accessing memory in a multi-strided manner enables more cache lines to be concurrently brought into the cache, resulting in improved cache hit ratios and higher effective memory bandwidth without the introduction of costly software prefetch instructions. Subsequently, we show that multi-strided variants of a collection of six memory-bound dense compute kernels outperform state-of-the-art counterparts on three different micro-architectures. More specifically, for kernels among which Matrix Vector Multiplication, Convolution Stencil and kernels from PolyBench, we achieve significant speedups of up to 12.55x over Polly, 2.99x over MKL, 1.98x over OpenBLAS, 1.08x over Halide and 1.87x over OpenCV. The code transformation to take advantage of multi-strided memory access is a natural extension of the loop unroll and loop interchange techniques, allowing this method to be incorporated into compiler pipelines in the future.

Figures (9)

• Figure 1: Illustration of the difference between a single-strided (left) and a multi-strided (right) access pattern, for two iterations, both using three unrolls.
• Figure 2: Measured throughput of different memory operations for increasing numbers of strides on the Core i7-8700 platform.
• Figure 3: Execution stalls with outstanding loads for L1 data, L2 and L3 cache misses, any outstanding load and the total number of stalls.
• Figure 4: Cache hit ratio for different cache levels with and without hardware prefetching.
• Figure 5: Throughput in gigibytes per second using different types of data-movement instructions for different stride configurations on exactly 2 GiB of data.