CAPIO: Safe Kernel-Bypass of Commodity Devices using Capabilities

TL;DR

CAP-IO introduces a capability-based approach to safe kernel-bypass I/O on commodity devices by overlaying CHERI hardware capabilities onto MMIO regions to create sub-page, bounded access tokens. By shifting enforcement from the MMU to hardware capabilities, CAP-IO decouples safety from device features, enabling real-world, low-latency user-space drivers for devices not designed for kernel bypass. The authors implement a proof-of-concept on the ARM Morello platform with an e1000e NIC and a minimal LwIP stack, achieving substantial latency reductions in microbenchmarks and in Memcached workloads. The work suggests broad applicability to MMIO devices and outlines future work toward NVMe and other privileged memory interfaces.

Abstract

Securing low-latency I/O in commodity systems forces a fundamental trade-off: rely on the kernel's high overhead mediated interface, or bypass it entirely, exposing sensitive hardware resources to userspace and creating new vulnerabilities. This dilemma stems from a hardware granularity mismatch: standard MMUs operate at page boundaries, making it impossible to selectively expose safe device registers without also exposing the sensitive control registers colocated on the same page. Existing solutions to driver isolation enforce an isolation model that cannot protect sub-page device resources. This paper presents CAPIO, the first architecture to leverage hardware capabilities to enforce fine-grained access control on memory-mapped I/O. Unlike prior page-based protections, CAPIO utilizes unforgeable capabilities to create precise, sub-page "slices" of device memory. This mechanism enables the kernel to delegate latency-critical hardware access to userspace applications while strictly preventing interaction with co-located privileged registers. We implement CAPIO based on CHERI on the ARM Morello platform and demonstrate a proof-of-concept safe-access driver for a commodity network card which was not originally designed for kernel bypass. We demonstrate that CAPIO achieves the latency improvements of kernel bypass while enforcing byte-level access control of privileged resources.

Figures (5)

• Figure 1: CAP-IO system architecture. The kernel generates bounded capabilities for the authorized driver, enabling direct access to device registers. A compromised or malicious application attempting to access privileged control registers on the same page is architecturally blocked by CHERI hardware bounds checks, preventing unauthorized device reconfiguration.
• Figure 2: CAP-IO implementation.
• Figure 3: When a CAP-IO-enabled kernel-bypass network driver is driven directly with UDP packets, tail (here, the 99th percentile tail) round-trip latencies are reduces substantially - up to 46% with packet sizes below 512 bytes. Higher numbers are better (lower latencies). These results are in-line with what is expected from kernel-bypass networking. The 46% latency reduction is a reduction of $73\mu{}s$ ($235\mu{}s \rightarrow 162\mu{}s$).
• Figure 4: With a full network stack (LWIP), CAP-IO-based kernel bypass networking still outperforms the FreeBSD kernel's stack, but the margin is less. This is an improvement of $44\mu{}s$ over the kernel in the top right configuration in the plot, implying the LwIP network stack has roughly $30\mu{}s$ overhead in that configuration.
• Figure 5: End-to-end latencies in Memcached driven by the YCSB workload generator have improvements across the board in TCP and UDP.