Ascendra: Dynamic Request Prioritization for Efficient LLM Serving

TL;DR

Ascendra tackles the challenge of efficient LLM serving by balancing throughput and latency under strict $TTFT$ and $TBT$ $SLO$s. It introduces a two-tier GPU architecture with Low-Priority (LP) and High-Priority (HP) instances and a deadline-aware, out-of-order prefill scheduler that proactively offloads urgent requests to HP. A regression-based performance model guides LP batching, while HP focuses on low-latency prefill, with a ticket-entry and elastic batching mechanism to maximize utilization. Evaluations across multiple models and datasets show Ascendra achieves significantly higher goodput and better $SLO$ adherence than baselines like $vLLM$ and Sarathi-Serve, without requiring expensive interconnects. The work demonstrates practical impact by enabling scalable, deadline-aware LLM serving on commodity GPU hardware.

Abstract

The rapid advancement of Large Language Models (LLMs) has driven the need for more efficient serving strategies. In this context, efficiency refers to the proportion of requests that meet their Service Level Objectives (SLOs), particularly for Time To First Token (TTFT) and Time Between Tokens (TBT). However, existing systems often prioritize one metric at the cost of the other. We present Ascendra, an LLM serving system designed to meet both TTFT and TBT SLOs simultaneously. The core insight behind Ascendra is that a request's urgency evolves as it approaches its deadline. To leverage this, Ascendra partitions GPU resources into two types of instances: low-priority and high-priority. Low-priority instances maximize throughput by processing requests out of arrival order, but at the risk of request starvation. To address this, Ascendra employs a performance model to predict requests at risk of missing their SLOs and proactively offloads them to high-priority instances. High-priority instances are optimized for low-latency execution and handle urgent requests nearing their deadlines. This partitioned architecture enables Ascendra to effectively balance high throughput and low latency. Extensive evaluation shows that Ascendra improves system throughput by up to 1.7x compared to vLLM and Sarathi-Serve while meeting both TTFT and TBT SLOs.

Figures (14)

• Figure 1: Top: Out-of-order scheduling and proactive offloading of Ascendra avoid SLO violations. Bottom: Ascendra improves SLO attainment and latency compared to baselines.
• Figure 2: SJF has lower latency compared to FCFS at the cost of a longer tail. Simply increasing the number of GPUs does not eliminate the tail.
• Figure 3: The prefill phase is compute-bound even with smaller batches, while the decode phase becomes memory-bound, utilizing only 60% of compute at a batch size of 128.
• Figure 4: Overview of Ascendra. Green arrows indicate the flow of non-urgent requests, while red arrows represent offloading from LP to HP instances. The controller routes incoming requests to LP instances by default, and to the HP instance when it is available or when a request is urgent.
• Figure 5: Detailed design of an LP instance. The system maintains separate queues for waiting prefill and ongoing decode requests. At each step, the scheduler uses a policy function and performance model to select requests from both queues and form a batch. It also flags requests at risk of SLO violation, placing them in an offloading queue for the controller to transfer to an HP instance. After execution, completed prefill requests are moved to the decode queue.