Rethinking Inter-Process Communication with Memory Operation Offloading

TL;DR

The paper tackles the IPC data-movement bottleneck in data-intensive workloads by proposing Rocket, a unified IPC runtime that orchestrates hardware memory offload (notably Intel DSA) with shared-memory IPC. It analyzes hardware/software integration challenges, designs a flexible IPC stack with sync, async, and pipelined execution modes, and demonstrates end-to-end benefits—up to $22\%$ fewer instructions, up to $2.1\times$ throughput, and up to $72\%$ latency reduction—across diverse benchmarks. Key contributions include system-level bottleneck analysis, a configurable protocol design, and comprehensive experimental evaluation on real hardware, illustrating when and how offload-aware IPC improves performance. The work suggests that treating memory offloading as a first-class system capability can yield tangible gains in data-intensive, intra-node contexts, guiding future hardware/software co-design for memory movement.

Abstract

As multimodal and AI-driven services exchange hundreds of megabytes per request, existing IPC runtimes spend a growing share of CPU cycles on memory copies. Although both hardware and software mechanisms are exploring memory offloading, current IPC stacks lack a unified runtime model to coordinate them effectively. This paper presents a unified IPC runtime suite that integrates both hardware- and software-based memory offloading into shared-memory communication. The system characterizes the interaction between offload strategies and IPC execution, including synchronization, cache visibility, and concurrency, and introduces multiple IPC modes that balance throughput, latency, and CPU efficiency. Through asynchronous pipelining, selective cache injection, and hybrid coordination, the system turns offloading from a device-specific feature into a general system capability. Evaluations on real-world workloads show instruction count reductions of up to 22%, throughput improvements of up to 2.1x, and latency reductions of up to 72%, demonstrating that coordinated IPC offloading can deliver tangible end-to-end efficiency gains in modern data-intensive systems.

Figures (15)

• Figure 1: Breakdown of end-to-end latency for intra-node echo RPCs implemented using shared memory (shmem) and gRPC. The figure quantifies the portion of total latency attributed to memcpy as a function of message size.
• Figure 2: DSA programming model. The CPU prepares the task descriptor and submits it to DSA. DSA executes the task and sets the completion flag. The CPU then checks the completion flag to determine if the task is complete.
• Figure 3: Comparison of polling strategies on latency and CPU usage (1MB transfer. lazypoll: polling every 100$\mu$s; busypoll: polling with yield but no sleep; UMWAIT: polling with usermode interrupt.)
• Figure 4: Performance comparison of DSA and CPU memcpy under different memory conditions. Copying to a pinned buffer reduces latency by 95%, and reusing the same buffer achieves a 97% reduction, both relative to cold-buffer access.
• Figure 5: LLC miss rates under single and four-threaded execution, comparing cpu_memcpy, dsa_memcpy, dsa_memcpy($inject) (Microbenchmark: summation over all elements in the destination buffer after memory copy).