Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips

Jiahuan Yu, Mingtao Hu, Zichao Lin, Minjia Zhang

TL;DR

SuperInfer tackles the memory-latency trade-off in LLM inference under tight TTFT and TBT SLOs by co-designing scheduling and memory-management for Superchips with NVLink-C2C. It introduces RotaSched, a proactive rotary scheduler guided by Virtual Lag Time ($VLT$), and DuplexKV, a full-duplex KV cache rotation engine that reorganizes KV layout and overlaps transfers with computation. The paper analyzes PCIe-based offloading limitations and demonstrates underutilization of NVLink-C2C in existing stacks, motivating the need for end-to-end co-design. Experimental results on GH200 show TTFT SLO attainment gains up to $74.7 rac{ ext{%}}{}$, with comparable throughput and TBT, verifying that SLO-aware scheduling and memory co-design unlocks the full potential of Superchips for responsive LLM serving.

Abstract

Large Language Model (LLM) serving faces a fundamental tension between stringent latency Service Level Objectives (SLOs) and limited GPU memory capacity. When high request rates exhaust the KV cache budget, existing LLM inference systems often suffer severe head-of-line (HOL) blocking. While prior work explored PCIe-based offloading, these approaches cannot sustain responsiveness under high request rates, often failing to meet tight Time-To-First-Token (TTFT) and Time-Between-Tokens (TBT) SLOs. We present SuperInfer, a high-performance LLM inference system designed for emerging Superchips (e.g., NVIDIA GH200) with tightly coupled GPU-CPU architecture via NVLink-C2C. SuperInfer introduces RotaSched, the first proactive, SLO-aware rotary scheduler that rotates requests to maintain responsiveness on Superchips, and DuplexKV, an optimized rotation engine that enables full-duplex transfer over NVLink-C2C. Evaluations on GH200 using various models and datasets show that SuperInfer improves TTFT SLO attainment rates by up to 74.7% while maintaining comparable TBT and throughput compared to state-of-the-art systems, demonstrating that SLO-aware scheduling and memory co-design unlocks the full potential of Superchips for responsive LLM serving.

SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips

TL;DR

SuperInfer tackles the memory-latency trade-off in LLM inference under tight TTFT and TBT SLOs by co-designing scheduling and memory-management for Superchips with NVLink-C2C. It introduces RotaSched, a proactive rotary scheduler guided by Virtual Lag Time (), and DuplexKV, a full-duplex KV cache rotation engine that reorganizes KV layout and overlaps transfers with computation. The paper analyzes PCIe-based offloading limitations and demonstrates underutilization of NVLink-C2C in existing stacks, motivating the need for end-to-end co-design. Experimental results on GH200 show TTFT SLO attainment gains up to , with comparable throughput and TBT, verifying that SLO-aware scheduling and memory co-design unlocks the full potential of Superchips for responsive LLM serving.

Abstract

Large Language Model (LLM) serving faces a fundamental tension between stringent latency Service Level Objectives (SLOs) and limited GPU memory capacity. When high request rates exhaust the KV cache budget, existing LLM inference systems often suffer severe head-of-line (HOL) blocking. While prior work explored PCIe-based offloading, these approaches cannot sustain responsiveness under high request rates, often failing to meet tight Time-To-First-Token (TTFT) and Time-Between-Tokens (TBT) SLOs. We present SuperInfer, a high-performance LLM inference system designed for emerging Superchips (e.g., NVIDIA GH200) with tightly coupled GPU-CPU architecture via NVLink-C2C. SuperInfer introduces RotaSched, the first proactive, SLO-aware rotary scheduler that rotates requests to maintain responsiveness on Superchips, and DuplexKV, an optimized rotation engine that enables full-duplex transfer over NVLink-C2C. Evaluations on GH200 using various models and datasets show that SuperInfer improves TTFT SLO attainment rates by up to 74.7% while maintaining comparable TBT and throughput compared to state-of-the-art systems, demonstrating that SLO-aware scheduling and memory co-design unlocks the full potential of Superchips for responsive LLM serving.
Paper Structure (27 sections, 1 equation, 25 figures, 1 table, 1 algorithm)

This paper contains 27 sections, 1 equation, 25 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. LLM Inference and Offloading over PCIe GPUs
  4. Offloading over Emerging Tightly Coupled GPU–CPU Architectures
  5. Scheduling for Latency-Aware Inference
  6. Observations
  7. SLO-Aware Offloading as a Challenge
  8. Responsiveness Bottleneck Caused by Low Swap Bandwidth over PCIe
  9. Superchip Offers Opportunity but Not for Free
  10. System Design
  11. Overall Architecture
  12. RotaSched: OS-Inspired Rotary Scheduler for SLO-Aware Offloading
  13. From Passive Preemption to Proactive Rotation
  14. Virtual Lag Time (VLT)
  15. Largest-VLT-First (LVF) Scheduling
  16. ...and 12 more sections

Figures (25)

  • Figure 1: Two static offloading policies: Waiting-First (WF) and Swapped-First (SF), and comparison of their P99 TTFT and TBT to FCFS under varying request rates (Qwen2.5-32B, ShareGPT).
  • Figure 2: P99 TTFT and TBT latency vs. swap bandwidth for vLLM with offloading. (Qwen2.5-32B, ShareGPT, RPS=20).
  • Figure 3: Limitations of low swap bandwidth for offloading: slow clearing of backlogged waiting requests and new HOL blocking in the swapped queue.
  • Figure 4: Measured bidirectional CPU-GPU KV cache transfer bandwidth of vLLM for three different models.
  • Figure 5: NVLink-C2C (GH200) vs. PCIe Gen5x16 (H200) full-duplex bandwidth by segment size.
  • ...and 20 more figures