Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

My CXL Pool Obviates Your PCIe Switch

Yuhong Zhong, Daniel S. Berger, Pantea Zardoshti, Enrique Saurez, Jacob Nelson, Antonis Psistakis, Joshua Fried, Asaf Cidon

TL;DR

This work addresses the inefficiency and high cost of PCIe switches by proposing a software-based PCIe pooling approach built on CXL memory pools. It introduces a two-part design comprising a datapath that routes PCIe traffic through shared CXL memory and a pooling orchestrator that dynamically allocates devices and migrates workloads, leveraging a cross-host shared-memory channel for fast signaling. The key contributions include empirical observations on memory-pool latency, a cross-host signaling mechanism with sub-microsecond latency, and a practical blueprint for deploying PCIe pooling today without hardware switches. If adopted, the approach could enable disaggregated NICs, accelerators, and other PCIe devices with lower costs and greater flexibility, while complementing RDMA-based storage disaggregation and enabling new datacenter architectures.

Abstract

Pooling PCIe devices across multiple hosts offers a promising solution to mitigate stranded I/O resources, enhance device utilization, address device failures, and reduce total cost of ownership. The only viable option today are PCIe switches, which decouple PCIe devices from hosts by connecting them through a hardware switch. However, the high cost and limited flexibility of PCIe switches hinder their widespread adoption beyond specialized datacenter use cases. This paper argues that PCIe device pooling can be effectively implemented in software using CXL memory pools. CXL memory pools improve memory utilization and already have positive return on investment. We find that, once CXL pools are in place, they can serve as a building block for pooling any kind of PCIe device. We demonstrate that PCIe devices can directly use CXL memory as I/O buffers without device modifications, which enables routing PCIe traffic through CXL pool memory. This software-based approach is deployable on today's hardware and is more flexible than hardware PCIe switches. In particular, we explore how disaggregating devices such as NICs can transform datacenter infrastructure.

My CXL Pool Obviates Your PCIe Switch

TL;DR

This work addresses the inefficiency and high cost of PCIe switches by proposing a software-based PCIe pooling approach built on CXL memory pools. It introduces a two-part design comprising a datapath that routes PCIe traffic through shared CXL memory and a pooling orchestrator that dynamically allocates devices and migrates workloads, leveraging a cross-host shared-memory channel for fast signaling. The key contributions include empirical observations on memory-pool latency, a cross-host signaling mechanism with sub-microsecond latency, and a practical blueprint for deploying PCIe pooling today without hardware switches. If adopted, the approach could enable disaggregated NICs, accelerators, and other PCIe devices with lower costs and greater flexibility, while complementing RDMA-based storage disaggregation and enabling new datacenter architectures.

Abstract

Pooling PCIe devices across multiple hosts offers a promising solution to mitigate stranded I/O resources, enhance device utilization, address device failures, and reduce total cost of ownership. The only viable option today are PCIe switches, which decouple PCIe devices from hosts by connecting them through a hardware switch. However, the high cost and limited flexibility of PCIe switches hinder their widespread adoption beyond specialized datacenter use cases. This paper argues that PCIe device pooling can be effectively implemented in software using CXL memory pools. CXL memory pools improve memory utilization and already have positive return on investment. We find that, once CXL pools are in place, they can serve as a building block for pooling any kind of PCIe device. We demonstrate that PCIe devices can directly use CXL memory as I/O buffers without device modifications, which enables routing PCIe traffic through CXL pool memory. This software-based approach is deployable on today's hardware and is more flexible than hardware PCIe switches. In particular, we explore how disaggregating devices such as NICs can transform datacenter infrastructure.

Paper Structure

This paper contains 24 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Why is PCIe Pooling a Good Idea?
  3. Stranded I/O Resources
  4. PCIe Device Failures
  5. CXL Memory Pools
  6. Latency and bandwidth.
  7. CXL memory pools.
  8. Shared CXL memory.
  9. Cache coherence.
  10. Designing PCIe Pools in Software
  11. PCIe Datapath Over CXL
  12. Placing buffers in shared CXL memory
  13. Performance implications of CXL
  14. Event signaling and host-to-host communications
  15. Pooling Orchestrator
  16. ...and 9 more sections

Figures (4)

  • Figure 1: A CXL memory pool enables hosts to access any PCIe device in a CXL pod, forming a logical pool of PCIe devices.
  • Figure 2: Percentages of stranded CPU cores, memory capacity, SSD storage, and NIC bandwidth in Microsoft Azure datacenters.
  • Figure 3: Latency-throughput graph of the UDP microbenchmark with 100 Gbps NICs. On the server side, TX and RX buffers are allocated either from the CXL memory pool (dotted lines) or local DDR5 memory (solid lines).
  • Figure 4: Latency distribution of message passing.