Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CXLRAMSim v1.0: System-Level Exploration of CXL Memory Expander Cards

Karan Pathak, David Atienza, Marina Zapater

Abstract

The growing demands in the training and inference of Large Language Models (LLMs) are accelerating the adoption of scale-up systems that extend server shared memory through the use of Compute Express Link (CXL)-based load/store interconnects. Accurate full-system simulation of such architectures remains challenging, as existing tools (all very recent) rely on simplified or non-compliant architectural models, impacting accuracy and usability. We present CXLRAMSim, the first gem5-integrated, full-system simulator that models CXL devices at their correct position on the I/O bus, enabling the use of unmodified Linux kernels and software stack, realistic latency-bandwidth behavior and true interleaving with system DRAM. Our approach provides high-fidelity CXL.mem characterization and captures key challenges such as cache pollution when accessing CXL memory.

CXLRAMSim v1.0: System-Level Exploration of CXL Memory Expander Cards

Abstract

The growing demands in the training and inference of Large Language Models (LLMs) are accelerating the adoption of scale-up systems that extend server shared memory through the use of Compute Express Link (CXL)-based load/store interconnects. Accurate full-system simulation of such architectures remains challenging, as existing tools (all very recent) rely on simplified or non-compliant architectural models, impacting accuracy and usability. We present CXLRAMSim, the first gem5-integrated, full-system simulator that models CXL devices at their correct position on the I/O bus, enabling the use of unmodified Linux kernels and software stack, realistic latency-bandwidth behavior and true interleaving with system DRAM. Our approach provides high-fidelity CXL.mem characterization and captures key challenges such as cache pollution when accessing CXL memory.

Paper Structure

This paper contains 10 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Simulation Framework
  5. Firmware Model
  6. CXL Hardware Model
  7. CXL.io protocol
  8. CXL.mem protocol
  9. Experiments
  10. Conclusion

Figures (5)

  • Figure 1: Full system simulators implementing CXL extension A) CXLDMSim and SimCXL on MemBusTCAD, HPCA and B) CXLRAMSim on IOBus.
  • Figure 2: Modeled X86 Bios in gem5 to support CXL2.0 devices
  • Figure 3: Modeled CXL Registers adhering to CXL2.0 specifications
  • Figure 4: CXL.mem protocol Transaction Layer implementation in gem5
  • Figure 5: L2 Miss rate for Stream micro-benchmark