Prefetching in Deep Memory Hierarchies with NVRAM as Main Memory

TL;DR

This work tackles the challenge of using NVRAM as main memory by evaluating a hybrid NV-S-D memory hierarchy and studying multi-level prefetching that spans both off-chip (HMC) and on-chip (L1) caches. It implements two prefetching architectures, HMC and HMC+L1, and shows that on an aggressive out-of-order processor, HMC alone can yield IPC improvements around 9%, while adding L1 prefetchers boosts gains to about 12%. Across Redis, Memtier, and MySQL workloads, the study reports high off-chip prefetching accuracy and coverage (up to 60–92%), significant reductions in L2/L1 MPKI, and substantial AMAT improvements (up to ~60% in some cases). The findings demonstrate that multi-level, cross-layer prefetching is essential to translating the benefits of HMC-based NVRAM main memory into tangible system-level performance gains, highlighting practical impact for data-centric workloads. The results motivate applying similar multi-level prefetching strategies to a broader set of hybrid memory configurations.

Abstract

Emerging applications, such as big data analytics and machine learning, require increasingly large amounts of main memory, often exceeding the capacity of current commodity processors built on DRAM technology. To address this, recent research has focused on off-chip memory controllers that facilitate access to diverse memory media, each with unique density and latency characteristics. While these solutions improve memory system performance, they also exacerbate the already significant memory latency. As a result, multi-level prefetching techniques are essential to mitigate these extended latencies. This paper investigates the advantages of prefetching across both sides of the memory system: the off-chip memory and the on-chip cache hierarchy. Our primary objective is to assess the impact of a multi-level prefetching engine on overall system performance. Additionally, we analyze the individual contribution of each prefetching level to system efficiency. To achieve this, the study evaluates two key prefetching approaches: HMC (Hybrid Memory Controller) and HMC+L1, both of which employ prefetching mechanisms commonly used by processor vendors. The HMC approach integrates a prefetcher within the off-chip hybrid memory controller, while the HMC+L1 approach combines this with additional L1 on-chip prefetchers. Experimental results on an out-of-order execution processor show that on-chip cache prefetchers are crucial for maximizing the benefits of off-chip prefetching, which in turn further enhances performance. Specifically, the off-chip HMC prefetcher achieves coverage and accuracy rates exceeding 60% and up to 80%, while the combined HMC+L1 approach boosts off-chip prefetcher coverage to as much as 92%. Consequently, overall performance increases from 9% with the HMC approach to 12% when L1 prefetching is also employed.

Figures (4)

• Figure 1: Hybrid memory system organization.
• Figure 2: Parameters used for each workload.
• Figure 3: Performance (coverage and accuracy) of the implemented prefetchers across the studied benchmarks.
• Figure 4: Memory hierarchy and overall performance metrics.