sNVMe-oF: Secure and Efficient Disaggregated Storage

TL;DR

Disaggregated storage with NVMe-oF in data centers faces confidential computing security requirements that existing methods struggle to meet at scale. sNVMe-oF extends NVMe-oF with a scalable control path, a metadata-driven data path, and a novel Hazel Merkle Tree to provide confidentiality, integrity, and freshness without modifying the protocol, while offloading work to CC-capable smart NICs. The approach leverages per-device IV/counter leasing, device-level key derivation via a KBS, and an asynchronous freshness verification path to achieve near line-rate performance with overheads as low as a few percent. A concrete prototype on NVIDIA BlueField-3 using SPDK/DOCA demonstrates practicality for PB-scale disaggregated CC storage, offering a viable path for secure, scalable data-center storage in the CC era.

Abstract

Disaggregated storage with NVMe-over-Fabrics (NVMe-oF) has emerged as the standard solution in modern data centers, achieving superior performance, resource utilization, and power efficiency. Simultaneously, confidential computing (CC) is becoming the de facto security paradigm, enforcing stronger isolation and protection for sensitive workloads. However, securing state-of-the-art storage with traditional CC methods struggles to scale and compromises performance or security. To address these issues, we introduce sNVMe-oF, a storage management system extending the NVMe-oF protocol and adhering to the CC threat model by providing confidentiality, integrity, and freshness guarantees. sNVMe-oF offers an appropriate control path and novel concepts such as counter-leasing. sNVMe-oF also optimizes data path performance by leveraging NVMe metadata, introducing a new disaggregated Hazel Merkle Tree (HMT), and avoiding redundant IPSec protections. We achieve this without modifying the NVMe-oF protocol. To prevent excessive resource usage while delivering line rate, sNVMe-oF also uses accelerators of CC-capable smart NICs. We prototype sNVMe-oF on an NVIDIA BlueField-3 and demonstrate how it can achieve as little as 2% performance degradation for synthetic patterns and AI training.

Figures (8)

• Figure 1: An overview of the locally deployed methods and sNVMe-oF with its novel contributions in scalability , algorithms , and resource efficiency .
• Figure 2: Read throughput, IOPS, and latency across different encryption and integrity algorithms with average CPU usage on a hyperthreaded system (50% CPU usage involves all physical cores). Overheads are relative to unprotected bare metal.
• Figure 3: An overview of sNVMe-oF's control flow with multiple stages of attestation, key derivation, and counter leasing.
• Figure 4: The pipeline of requests and how sNVMe-oF creates an encapsulation at each layer to provide security. The SSD zoom-in shows the layout of sectors that sNVMe-oF applies to the underlying SSDs.
• Figure 5: Performance of sNVMe-oF across different read patterns.