Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference

TL;DR

Harvest addresses memory bottlenecks in LLM inference by using opportunistic peer-GPU memory, connected over NVLink, as a best-effort cache for model weights and KV cache. It introduces a three-tier memory model, a lightweight API/runtime, and deployment mechanisms (including MIG isolation) to migrate data to peer memory when available without changing model code. Across MoE and KV cache workloads, Harvest achieves significant gains: up to approximately 10x reduction in expert transfer latency and about 1.5–2.0x end-to-end throughput gains, with KV cache transfers improving by up to around 5.65x compared with CPU offload. The approach reduces reliance on PCIe-based paging, improves utilization of multi-GPU servers, and provides a practical path to scaling inference throughput under memory pressure while preserving correctness through explicit durability strategies.

Abstract

Large Language Model (LLM) inference is increasingly constrained by GPU memory capacity rather than compute throughput, driven by growing model sizes and the linear growth of the key-value (KV) cache during autoregressive decoding. Existing approaches mitigate memory pressure by offloading model state and KV tensors to host memory, but incur substantial latency due to limited PCIe bandwidth. We present Harvest, an opportunistic GPU cache management framework that exploits high-bandwidth peer-to-peer GPU interconnects to dynamically place model weights and KV cache in unused GPU memory. Harvest treats peer GPU memory as a transient cache tier, preserving correctness while reducing data movement overhead under dynamic memory availability. We demonstrate significant throughput speedup of more than 2 times by using Harvest to accelerate the retrieval of two widely-used inference components: expert layer weights and KV cache entries.

Figures (7)

• Figure 1: Harvest enables the use of fast remote NVLink-connected GPU memory as an opportunistic cache.
• Figure 2: CDF of GPU Memory Consumption across Alibaba Cloud Training and Inference Cluster.
• Figure 3: GPU$\leftrightarrow$GPU and GPU$\leftrightarrow$CPU transfer latency of memory chunks of different sizes, mapped to expert sizes of different MoE models for reference.
• Figure 4: Harvest architecture. The Harvest runtime spans the compute and peer GPUs under the same NVLink domain and responds to application-initiated memory allocation requests by harvesting available fast memory in a peer GPU. If the allocation fails or is revoked, the application can revert to the slow host DRAM.
• Figure 5: Token generation throughput improvement with expert weight offloading to peer GPU using Harvest vs to CPU using CGOPipe. In this scenario, we force 50% experts to be offloaded.