Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DOLMA: A Data Object Level Memory Disaggregation Framework for HPC Applications

Haoyu Zheng, Shouwei Gao, Jie Ren, Wenqian Dong

TL;DR

HPC centers underutilize memory, motivating memory disaggregation. DOLMA introduces a data object-level framework atop RDMA that offloads large, long-lived objects to remote memory while keeping hot objects local, enabling dual-buffered remote reads and asynchronous writes to hide latency. The approach includes a principled data-object selection strategy, memory-region partitioning, and multi-threading accommodations, implemented with one-sided RDMA. Evaluation on eight HPC workloads shows up to 63% local-memory reduction and less than 16% degradation relative to a fully local baseline, demonstrating practical memory-disaggregation potential for HPC workloads.

Abstract

Memory disaggregation is promising to scale memory capacity and improves utilization in HPC systems. However, the performance overhead of accessing remote memory poses a significant challenge, particularly for compute-intensive HPC applications where execution times are highly sensitive to data locality. In this work, we present DOLMA, a Data Object Level M emory dis Aggregation framework designed for HPC applications. DOLMA intelligently identifies and offloads data objects to remote memory, while providing quantitative analysis to decide a suitable local memory size. Furthermore, DOLMA leverages the predictable memory access patterns typical in HPC applications and enables remote memory prefetch via a dual-buffer design. By carefully balancing local and remote memory usage and maintaining multi-thread concurrency, DOLMA provides a flexible and efficient solution for leveraging disaggregated memory in HPC domains while minimally compromising application performance. Evaluating with eight HPC workloads and computational kernels, DOLMA limits performance degradation to less than 16% while reducing local memory usage by up to 63%, on average.

DOLMA: A Data Object Level Memory Disaggregation Framework for HPC Applications

TL;DR

HPC centers underutilize memory, motivating memory disaggregation. DOLMA introduces a data object-level framework atop RDMA that offloads large, long-lived objects to remote memory while keeping hot objects local, enabling dual-buffered remote reads and asynchronous writes to hide latency. The approach includes a principled data-object selection strategy, memory-region partitioning, and multi-threading accommodations, implemented with one-sided RDMA. Evaluation on eight HPC workloads shows up to 63% local-memory reduction and less than 16% degradation relative to a fully local baseline, demonstrating practical memory-disaggregation potential for HPC workloads.

Abstract

Memory disaggregation is promising to scale memory capacity and improves utilization in HPC systems. However, the performance overhead of accessing remote memory poses a significant challenge, particularly for compute-intensive HPC applications where execution times are highly sensitive to data locality. In this work, we present DOLMA, a Data Object Level M emory dis Aggregation framework designed for HPC applications. DOLMA intelligently identifies and offloads data objects to remote memory, while providing quantitative analysis to decide a suitable local memory size. Furthermore, DOLMA leverages the predictable memory access patterns typical in HPC applications and enables remote memory prefetch via a dual-buffer design. By carefully balancing local and remote memory usage and maintaining multi-thread concurrency, DOLMA provides a flexible and efficient solution for leveraging disaggregated memory in HPC domains while minimally compromising application performance. Evaluating with eight HPC workloads and computational kernels, DOLMA limits performance degradation to less than 16% while reducing local memory usage by up to 63%, on average.

Paper Structure

This paper contains 24 sections, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Disaggregated Memory and HPC Workloads
  4. One-side RDMA Communication
  5. Preliminary Study and Motivation
  6. Characteristics of Remote Memory Access
  7. Data Objects in HPC Applications
  8. Design
  9. Remote Data Objects Selection
  10. Memory Management of DOLMA
  11. Handling Multi-threading within One Compute Node
  12. Implementation
  13. Evaluation
  14. Overall Performance
  15. Execution Time Analysis
  16. ...and 9 more sections

Figures (10)

  • Figure 1: Disaggregated memory architecture
  • Figure 2: Architectures of different disaggregrated memory systems.
  • Figure 3: Example of one-side RDMA access.
  • Figure 4: Normalized performance of remote memory accesses with various data object sizes. The performance is normalized to local memory accesses (oracle), and the remote access latency via Infiniband (in microseconds) is labeled above the red bar.
  • Figure 5: Statistic of data objects in HPC application Laghos.
  • ...and 5 more figures