Asynchronous Memory Access Unit: Exploiting Massive Parallelism for Far Memory Access

TL;DR

This work tackles the memory wall exacerbated by far memory latency by introducing AMI and AMU, a hardware-software cooperative solution that exposes massive memory-level parallelism inside a contemporary OoO core. By using a portion of the L2 cache as Scratchpad Memory (SPM) and decoupling memory request issuance from response handling, AMU supports hundreds of in-flight memory operations, with a coroutine-based framework enabling efficient software scheduling. The approach demonstrates substantial performance gains on memory-bound workloads (average 2.42x at 1µs latency, up to 26.86x on GUPS at 5µs) and maintains scalable MLP as far-memory latency increases, while keeping hardware overhead modest through reuse of existing structures and software-based memory disambiguation. The results indicate that asynchronous memory access inside the core can effectively hide far memory latency and enable practical, high-MLP execution for data-center workloads, with future work spanning OS/compiler support and broader memory-pattern extensions.

Abstract

The growing memory demands of modern applications have driven the adoption of far memory technologies in data centers to provide cost-effective, high-capacity memory solutions. However, far memory presents new performance challenges because its access latencies are significantly longer and more variable than local DRAM. For applications to achieve acceptable performance on far memory, a high degree of memory-level parallelism (MLP) is needed to tolerate the long access latency. While modern out-of-order processors are capable of exploiting a certain degree of MLP, they are constrained by resource limitations and hardware complexity. The key obstacle is the synchronous memory access semantics of traditional load/store instructions, which occupy critical hardware resources for a long time. The longer far memory latencies exacerbate this limitation. This paper proposes a set of Asynchronous Memory Access Instructions (AMI) and its supporting function unit, Asynchronous Memory Access Unit (AMU), inside a contemporary Out-of-Order Core. AMI separates memory request issuing from response handling to reduce resource occupation. Additionally, AMU architecture supports up to several hundreds of asynchronous memory requests through re-purposing a portion of L2 Cache as scratchpad memory (SPM) to provide sufficient temporal storage. Together with a coroutine-based programming framework, this scheme can achieve significantly higher MLP for hiding far memory latencies. Evaluation with a cycle-accurate simulation shows AMI achieves 2.42x speedup on average for memory-bound benchmarks with 1us additional far memory latency. Over 130 outstanding requests are supported with 26.86x speedup for GUPS (random access) with 5 us latency. These demonstrate how the techniques tackle far memory performance impacts through explicit MLP expression and latency adaptation.

Figures (11)

• Figure 1: Far Memory in Emerging Data Center Servers
• Figure 2: Benchmark slowdown under different far memory latencies. The Y-axis shows the normalized performance slowdown relative to the performance under 100 ns far memory access latency.
• Figure 3: The performance of GUPS as hardware resources are increased. The CXL Ideal configuration is described in Section \ref{['sec:evaluation']}. The hardware resources (i.e. IQ, LSQ, ROB, MSHRs, and physical registers) of x2 and x4 are two times and four times those of CXL Ideal configuration, respectively. The lines show the performance of Group Prefetching chen2007improving based GUPS under different group (or batch) sizes. The bar plot represents the performance of the baseline GUPS benchmark.
• Figure 4: AMU Architecture
• Figure 5: Architecture of ALSU