Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CXL Shared Memory Programming: Barely Distributed and Almost Persistent

Yi Xu, Suyash Mahar, Ziheng Liu, Mingyao Shen, Steven Swanson

TL;DR

This paper addresses the absence of a formal failure model for CXL-based shared memory by defining independent data-failure and process-failure domains and arguing that traditional distributed-system techniques must be adapted for CXL's load/store semantics. It introduces a three-domain failure model and proposes a suite of tailored mitigation strategies, including CXL switch-based replication, erasure coding with grouping, and whole-system persistence inspired by PMEM concepts. To handle failures, the authors explore both application-dependent (undo/redo/resumption logs) and application-independent (checkpointing, whole-process persistence) methods, emphasizing metadata placement and co-design to balance latency, throughput, and memory overhead. They further present layered and co-designed approaches to mitigate data and process failures together, such as redo-compacted replication, stage-based replication, and log-structured memory on checkpoints, with concrete performance trade-offs. Overall, the work lays a foundational framework for resilient, disaggregated CXL memory systems and highlights practical benefits for process migration and data access bandwidth.

Abstract

While Compute Express Link (CXL) enables support for cache-coherent shared memory among multiple nodes, it also introduces new types of failures--processes can fail before data does, or data might fail before a process does. The lack of a failure model for CXL-based shared memory makes it challenging to understand and mitigate these failures. To solve these challenges, in this paper, we describe a model categorizing and handling the CXL-based shared memory's failures: data and process failures. Data failures in CXL-based shared memory render data inaccessible or inconsistent for a currently running application. We argue that such failures are unlike data failures in distributed storage systems and require CXL-specific handling. To address this, we look into traditional data failure mitigation techniques like erasure coding and replication and propose new solutions to better handle data failures in CXL-based shared memory systems. Next, we look into process failures and compare the failures and potential solutions with PMEM's failure model and programming solutions. We argue that although PMEM shares some of CXL's characteristics, it does not fully address CXL's volatile nature and low access latencies. Finally, taking inspiration from PMEM programming solutions, we propose techniques to handle these new failures. Thus, this paper is the first work to define the CXL-based shared memory failure model and propose tailored solutions that address challenges specific to CXL-based systems.

CXL Shared Memory Programming: Barely Distributed and Almost Persistent

TL;DR

This paper addresses the absence of a formal failure model for CXL-based shared memory by defining independent data-failure and process-failure domains and arguing that traditional distributed-system techniques must be adapted for CXL's load/store semantics. It introduces a three-domain failure model and proposes a suite of tailored mitigation strategies, including CXL switch-based replication, erasure coding with grouping, and whole-system persistence inspired by PMEM concepts. To handle failures, the authors explore both application-dependent (undo/redo/resumption logs) and application-independent (checkpointing, whole-process persistence) methods, emphasizing metadata placement and co-design to balance latency, throughput, and memory overhead. They further present layered and co-designed approaches to mitigate data and process failures together, such as redo-compacted replication, stage-based replication, and log-structured memory on checkpoints, with concrete performance trade-offs. Overall, the work lays a foundational framework for resilient, disaggregated CXL memory systems and highlights practical benefits for process migration and data access bandwidth.

Abstract

While Compute Express Link (CXL) enables support for cache-coherent shared memory among multiple nodes, it also introduces new types of failures--processes can fail before data does, or data might fail before a process does. The lack of a failure model for CXL-based shared memory makes it challenging to understand and mitigate these failures. To solve these challenges, in this paper, we describe a model categorizing and handling the CXL-based shared memory's failures: data and process failures. Data failures in CXL-based shared memory render data inaccessible or inconsistent for a currently running application. We argue that such failures are unlike data failures in distributed storage systems and require CXL-specific handling. To address this, we look into traditional data failure mitigation techniques like erasure coding and replication and propose new solutions to better handle data failures in CXL-based shared memory systems. Next, we look into process failures and compare the failures and potential solutions with PMEM's failure model and programming solutions. We argue that although PMEM shares some of CXL's characteristics, it does not fully address CXL's volatile nature and low access latencies. Finally, taking inspiration from PMEM programming solutions, we propose techniques to handle these new failures. Thus, this paper is the first work to define the CXL-based shared memory failure model and propose tailored solutions that address challenges specific to CXL-based systems.
Paper Structure (20 sections, 2 figures)

This paper contains 20 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Failure Model
  3. Handling Failures in CXL-based System
  4. Data Failure
  5. Process Failure
  6. Background of Data Consistency.
  7. Application-dependent Methods.
  8. Application-independent Methods.
  9. Key Insights.
  10. Handling Process and Data Failure Together
  11. Layered Process and Data Failure Handling
  12. Replication/Erasure Coding with Undo Logging.
  13. Replication/Erasure Coding with Atomic Pointer Updates.
  14. Replication/Erasure Coding with Whole Process Persistence.
  15. Co-designed Process and Data Failure Handling
  16. ...and 5 more sections

Figures (2)

  • Figure 1: CXL's Failure Model. Three failure scenarios based on the application's location and data. (1) Domain #0 with data: data failure. (2) Domain #1 with execution context: process failure. (3) Domain #2 with data and execution context: data and process failure.
  • Figure 2: Failure Models of Traditional Systems.