Guidelines for Building Indexes on Partially Cache-Coherent CXL Shared Memory

TL;DR

The paper addresses the challenge of building correct and efficient indexes on Partial Cache-Coherence (PCC) platforms, where full hardware cache coherence is relaxed to scale memory sharing across hosts. It presents SP guidelines to convert existing CCIndexes to PCCIndex by classifying data into sync-data and protected-data and applying cache-bypass primitives appropriately, alongside crash-consistency techniques. To overcome performance bottlenecks, it introduces P3 guidelines—Out-of-place updates, Replicated shared variables, and Speculative Reading—and demonstrates substantial throughput improvements in CLevelHash and BwTree, including real-world traces and Ray integration. The results show that SP$^4$ plus P$^3$ optimizations can yield up to 16x throughput gains over unoptimized PCCIndexes and outperform RDMA-based or partitioned approaches by significant factors, making PCCIndex deployment practical for large-scale, cross-host memory systems.

Abstract

The \emph{Partial Cache-Coherence (PCC)} model maintains hardware cache coherence only within subsets of cores, enabling large-scale memory sharing with emerging memory interconnect technologies like Compute Express Link (CXL). However, PCC's relaxation of global cache coherence compromises the correctness of existing single-machine software. This paper focuses on building consistent and efficient indexes on PCC platforms. We present that existing indexes designed for cache-coherent platforms can be made consistent on PCC platforms following SP guidelines, i.e., we identify \emph{sync-data} and \emph{protected-data} according to the index's concurrency control mechanisms, and synchronize them accordingly. However, conversion with SP guidelines introduces performance overhead. To mitigate the overhead, we identify several unique performance bottlenecks on PCC platforms, and propose P$^3$ guidelines (i.e., using Out-of-\underline{P}lace update, Re\underline{P}licated shared variable, S\underline{P}eculative Reading) to improve the efficiency of converted indexes on PCC platforms. With SP and P$^3$ guidelines, we convert and optimize two indexes (CLevelHash and BwTree) for PCC platforms. Evaluation shows that converted indexes' throughput improves up to 16$\times$ following P$^3$ guidelines, and the optimized indexes outperform their message-passing-based and disaggregated-memory-based counterparts by up to 16$\times$ and 19$\times$.

Figures (35)

• Figure 1: Partial cache-coherence model.
• Figure 2: Operations' behavior on PCC platforms.
• Figure 3: Existing indexes. (a), (c) and (e) employs a share-everything architecture, while (b) and (d) are share-nothing.
• Figure 4: Lock-based
• Figure 5: Lock-free