Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures

Evangelos Georganas, Dhiraj Kalamkar, Kirill Voronin, Abhisek Kundu, Antonio Noack, Hans Pabst, Alexander Breuer, Alexander Heinecke

TL;DR

This work introduces a framework to develop efficient, portable DL and HPC kernels for modern CPU architectures using Tensor Processing Primitives (TPPs), and demonstrates the efficacy of the approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.

Abstract

During the past decade, Deep Learning (DL) algorithms, programming systems and hardware have converged with the High Performance Computing (HPC) counterparts. Nevertheless, the programming methodology of DL and HPC systems is stagnant, relying on highly-optimized, yet platform-specific and inflexible vendor-optimized libraries. Such libraries provide close-to-peak performance on specific platforms, kernels and shapes thereof that vendors have dedicated optimizations efforts, while they underperform in the remaining use-cases, yielding non-portable codes with performance glass-jaws. This work introduces a framework to develop efficient, portable DL and HPC kernels for modern CPU architectures. We decompose the kernel development in two steps: 1) Expressing the computational core using Tensor Processing Primitives (TPPs): a compact, versatile set of 2D-tensor operators, 2) Expressing the logical loops around TPPs in a high-level, declarative fashion whereas the exact instantiation (ordering, tiling, parallelization) is determined via simple knobs. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.

Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures

TL;DR

This work introduces a framework to develop efficient, portable DL and HPC kernels for modern CPU architectures using Tensor Processing Primitives (TPPs), and demonstrates the efficacy of the approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.

Abstract

During the past decade, Deep Learning (DL) algorithms, programming systems and hardware have converged with the High Performance Computing (HPC) counterparts. Nevertheless, the programming methodology of DL and HPC systems is stagnant, relying on highly-optimized, yet platform-specific and inflexible vendor-optimized libraries. Such libraries provide close-to-peak performance on specific platforms, kernels and shapes thereof that vendors have dedicated optimizations efforts, while they underperform in the remaining use-cases, yielding non-portable codes with performance glass-jaws. This work introduces a framework to develop efficient, portable DL and HPC kernels for modern CPU architectures. We decompose the kernel development in two steps: 1) Expressing the computational core using Tensor Processing Primitives (TPPs): a compact, versatile set of 2D-tensor operators, 2) Expressing the logical loops around TPPs in a high-level, declarative fashion whereas the exact instantiation (ordering, tiling, parallelization) is determined via simple knobs. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
Paper Structure (31 sections, 11 figures, 2 tables)

This paper contains 31 sections, 11 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The PARLOOPER Framework
  3. GEMM Example Notations
  4. Logical Loop Declaration
  5. Expressing the Computation
  6. Auto-tuning the nested loops
  7. Performance modeling of generated loops
  8. Kernels Developed via PARLOOPER/TPP
  9. GEMM and Multi-Layer Perceptron (MLP)
  10. Extending GEMM with Activation Functions to MLP
  11. Extending the TPP backend with AArch64 SVE support
  12. Convolution Kernels
  13. Block-Sparse $\times$ Dense Matrix Multiply (Block-SpMM)
  14. DL Workloads via PARLOOPER/TPP
  15. Large Language Model Training and Inference
  16. ...and 16 more sections

Figures (11)

  • Figure 1: PARLOOPER & TPP framework workflow
  • Figure 2: GEMM performance of varying sizes on (Top) SPR, (Middle) Graviton 3, and (Bottom) Zen4
  • Figure 3: MLP with Bias-Add and RELU activations.
  • Figure 4: FP32 GEMM performance on SPR with PARLOOPER, oneDNN and TVM-Autoscheduler.
  • Figure 5: GEMM with sizes from BERT, GPT, DLRM modular
  • ...and 6 more figures