Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UPMEM Unleashed: Software Secrets for Speed

Krystian Chmielewski, Jarosław Ławnicki, Uladzislau Lukyanau, Tadeusz Kobus, Maciej Maciejewski

TL;DR

This work investigates the UPMEM Processing-in-Memory platform, identifies critical inefficiencies in compiler-generated code and NUMA-awareness, and demonstrates software-level optimizations that unlock substantial performance gains. By reworking arithmetic kernels (including INT8/INT32 and decomposed multiplication), employing loop unrolling, and introducing bit-serial processing for low-precision data, the authors achieve up to 1.6–2× addition speedups and 1.4–5.9× multiplication speedups, with BSDP yielding 2.7× improvements over baselines. They also implement NUMA-aware host–PIM data transfers, boosting throughput up to 2.9× and stabilizing performance. In memory-bound GEMV workloads, preloading matrices into PIM enables UPMEM to exceed a dual-socket CPU server by over 3× for INT8 and by 10× for INT4, validating the practical potential of optimized PIM kernels for AI and data-intensive tasks. Collectively, the results provide a concrete roadmap for making UPMEM a robust, predictable, and high-performance platform for PIM research and deployment.

Abstract

Developing kernels for Processing-In-Memory (PIM) platforms poses unique challenges in data management and parallel programming on limited processing units. Although software development kits (SDKs) for PIM, such as the UPMEM SDK, provide essential tools, these emerging platforms still leave significant room for performance optimization. In this paper, we reveal surprising inefficiencies in UPMEM software stack and play with non-standard programming techniques. By making simple modifications to the assembly generated by the UPMEM compiler, we achieve speedups of 1.6-2x in integer addition and 1.4-5.9x in integer multiplication, depending on the data type. We also demonstrate that bit-serial processing of low precision data is a viable option for UPMEM: in INT4 bit-serial dot-product calculation, UPMEM can achieve over 2.7x speedup over the baseline. Minor API extensions for PIM allocation that account for the non-uniform memory access (NUMA) architecture of the server further improve the consistency and throughput of host-PIM data transfers by up to 2.9x. Finally, we show that, when the matrix is preloaded into PIM, our optimized kernels outperform a dual-socket CPU server by over 3x for INT8 generalized matrix-vector multiplication (GEMV) and by 10x for INT4 GEMV. Our optimized INT8 GEMV kernel outperforms the baseline 3.5x.

UPMEM Unleashed: Software Secrets for Speed

TL;DR

This work investigates the UPMEM Processing-in-Memory platform, identifies critical inefficiencies in compiler-generated code and NUMA-awareness, and demonstrates software-level optimizations that unlock substantial performance gains. By reworking arithmetic kernels (including INT8/INT32 and decomposed multiplication), employing loop unrolling, and introducing bit-serial processing for low-precision data, the authors achieve up to 1.6–2× addition speedups and 1.4–5.9× multiplication speedups, with BSDP yielding 2.7× improvements over baselines. They also implement NUMA-aware host–PIM data transfers, boosting throughput up to 2.9× and stabilizing performance. In memory-bound GEMV workloads, preloading matrices into PIM enables UPMEM to exceed a dual-socket CPU server by over 3× for INT8 and by 10× for INT4, validating the practical potential of optimized PIM kernels for AI and data-intensive tasks. Collectively, the results provide a concrete roadmap for making UPMEM a robust, predictable, and high-performance platform for PIM research and deployment.

Abstract

Developing kernels for Processing-In-Memory (PIM) platforms poses unique challenges in data management and parallel programming on limited processing units. Although software development kits (SDKs) for PIM, such as the UPMEM SDK, provide essential tools, these emerging platforms still leave significant room for performance optimization. In this paper, we reveal surprising inefficiencies in UPMEM software stack and play with non-standard programming techniques. By making simple modifications to the assembly generated by the UPMEM compiler, we achieve speedups of 1.6-2x in integer addition and 1.4-5.9x in integer multiplication, depending on the data type. We also demonstrate that bit-serial processing of low precision data is a viable option for UPMEM: in INT4 bit-serial dot-product calculation, UPMEM can achieve over 2.7x speedup over the baseline. Minor API extensions for PIM allocation that account for the non-uniform memory access (NUMA) architecture of the server further improve the consistency and throughput of host-PIM data transfers by up to 2.9x. Finally, we show that, when the matrix is preloaded into PIM, our optimized kernels outperform a dual-socket CPU server by over 3x for INT8 generalized matrix-vector multiplication (GEMV) and by 10x for INT4 GEMV. Our optimized INT8 GEMV kernel outperforms the baseline 3.5x.
Paper Structure (21 sections, 6 equations, 13 figures, 2 algorithms)

This paper contains 21 sections, 6 equations, 13 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. UPMEM PIM architecture
  3. UPMEM Computational Performance
  4. Benchmark and baseline results
  5. Optimizing 8-bit Integer Multiplication
  6. Optimizing 32-bit Integer Multiplication
  7. Loop unrolling
  8. Bit-serial dot product
  9. Principles of bit-serial dot product
  10. Bit-serial dot product on UPMEM
  11. Performance results
  12. Data transfer optimizations
  13. Bottlenecks in host-PIM data transfers
  14. UPMEM SDK extensions
  15. Performance results
  16. ...and 6 more sections

Figures (13)

  • Figure 1: Overview of the UPMEM platform architecture
  • Figure 2: A microbenchmark used to evaluate the arithmetic performance of a single UPMEM DPU. Adapted from GEI+22. BLOCK_SIZE is set to 1024.
  • Figure 3: The baseline arithmetic performance of a single DPU. The performance levels off for 11 tasklets, because only 11 out of the 14 pipeline stages can operate concurrently.
  • Figure 4: The __mulsi3 routine used on UPMEM to perform INT32 multiplication.
  • Figure 5: Loading eight INT8 values as a single 64b block and multiplying them by a scalar using the correct UPMEM instructions.
  • ...and 8 more figures