Global Optimizations & Lightweight Dynamic Logic for Concurrency

TL;DR

GOLDYLOC introduces a lightweight dynamic logic which considers the dynamic execution environment for available parallelism and input sizes to execute performant combinations of concurrent GEMMs on the GPU and improves performance of concurrent GEMMs on a real GPU.

Abstract

Modern accelerators like GPUs are increasingly executing independent operations concurrently to improve the device's compute utilization. However, effectively harnessing it on GPUs for important primitives such as general matrix multiplications (GEMMs) remains challenging. Although modern GPUs have significant hardware and software support for GEMMs, their kernel implementations and optimizations typically assume each kernel executes in isolation and can utilize all GPU resources. This approach is highly efficient when kernels execute in isolation, but causes significant resource contention and slowdowns when kernels execute concurrently. Moreover, current approaches often only statically expose and control parallelism within an application, without considering runtime information such as varying input size and concurrent applications -- often exacerbating contention. These issues limit performance benefits from concurrently executing independent operations. Accordingly, we propose GOLDYLOC, which considers the global resources across all concurrent operations to identify performant GEMM kernels, which we call globally optimized (GO)-Kernels. Moreover, GOLDYLOC introduces a lightweight dynamic logic which considers the dynamic execution environment for available parallelism and input sizes to execute performant combinations of concurrent GEMMs on the GPU. Overall, GOLDYLOC improves performance of concurrent GEMMs on a real GPU by up to 2$\times$ (18% geomean per workload) and provides up to 2.5$\times$ (43% geomean per workload) speedups over sequential execution.

Figures (15)

• Figure 1: (a) Toy DNN computation. (b) High-level GEMM implementation on a GPU.
• Figure 2: ML algorithms with independent operations.
• Figure 3: (a) GEMM sizes with fewer FLOPs benefit less from concurrency (b) GEMM sizes with the same FLOPs can have different concurrency behavior. X-axis represent GEMMs as M_N_K_T1_T2 and GEMM FLOPs are calculated as 2*M*N*K.
• Figure 4: GEMM behavior with different kernel implementations. Kernels-1 and -2 are the GEMMs' isolated tuned kernels; Kernels-3 and -4 are alternate implementations with smaller memory traffic and fewer WG waves, respectively.
• Figure 5: (a) Speedups over sequential execution for 2 & 16 concurrent GEMMs (2P & 16P) versus the #waves in their isolated execution. (b) Speedups of GEMMs with fixed #waves but with varying K, input shape, or transpose.