Designing Transport-Level Encryption for Datacenter Networks

TL;DR

This work introduces SMT, a secure, transport-level encryption framework for datacenter networks that integrates TLS-based encryption with a native, message-based transport (initially Homa/NDP) to support RPC workloads. SMT preserves TLS security properties while enabling per-message record sequences and unordered message delivery, leveraging NIC offloads and TSO-compatible framing. The authors implement SMT as a Linux kernel patch to Homa/Linux and demonstrate up to 41% throughput gains and up to 35% latency reductions compared with TLS/TCP, along with favorable results for Redis and NVMe-oF workloads. A thorough analysis covers threat models, key exchange mechanisms (including 0-RTT via SMT-tickets), and security properties, arguing that SMT offers a practical path to replacing TLS/TCP in datacenters without sacrificing hardware offload or security guarantees. The work also discusses trade-offs, limitations, and future directions toward broader applicability to other datacenter transports like NDP and beyond.

Abstract

Cloud applications need network data encryption to isolate from other tenants and protect their data from potential eavesdroppers in the network infrastructure. This paper presents SMT, a protocol design for emerging datacenter transport protocols, such as NDP and Homa, to integrate data encryption. SMT integrates TLS-based encryption with a message-based transport protocol that supports efficient Remote Procedure Calls (RPCs), a common workload in datacenters. This architecture enables the use of per-message record sequence number spaces in a secure session, while ensuring unique message identities to prevent replay attacks. It also enables the use of existing NIC offloads designed for TLS over TCP, while being a native transport protocol alongside TCP and UDP. We implement SMT in the Linux kernel by extending Homa/Linux and improve RPC throughput by up to 41 % and latency by up to 35 % in comparison to TLS/TCP.

Figures (13)

• Figure 1: Key properties of encrypted or message-based transport methods (discussed in \ref{['sec:options:enc']} and \ref{['sec:options:msg']}).
• Figure 2: Generalized message-based transport packet format based on Homa homalinux and MTP mtp. Shaded parts are identical between the packets that belong to the same message. Msg off identifies the position of this packet within the message.
• Figure 3: Encryption with autonomous offload autonomous. Each rectangle represents one TLS record that contains one or more packets or TSO segments. The HW expects S2 after S1 to produce a correct next encrypted segment (In-seq); if S3 arrives, it generates a corrupted one (Out-seq.). A resync descriptor (R3) changes the seqno that HW expects to S3 (Out-resync). Note that each segment in the Wire actually consists of multiple packets split by TSO.
• Figure 4: SMT TSO segment with one TLS record being split to 3 packets. Dark and light gray parts overlay TCP common header and options space, respectively, and are replicated over every packet by TSO. The NIC encrypts the dashed area. TLS record header is actually 5 and the authentication tag is 16.
• Figure 5: Use of record sequence numbers across TCP/TLS, SMT, and QUIC-TLS rfc9001: TCP/TLS uses the 64-bit record sequence number; SMT encodes message ID and intra-message record index (\ref{['sec:design:msgid']}); QUIC uses the packet number (\ref{['sec:other']}).