Niyama : Breaking the Silos of LLM Inference Serving

TL;DR

The paper tackles inefficiencies in LLM inference serving caused by siloed, coarse-grained QoS deployments. It proposes Niyama, a QoS-aware LLM inference system that co-schedules multiple QoS classes on shared GPUs using dynamic chunking, a hybrid prioritization policy, and eager relegation. Across diverse models and datasets, Niyama demonstrates up to 32% higher serving capacity and an order-of-magnitude reduction in SLO violations under overload, along with substantial improvements in resource utilization. The work offers a practical path to fine-grained QoS differentiation in production LLM serving, enabling more efficient and reliable multi-tenant deployments.

Abstract

The widespread adoption of Large Language Models (LLMs) has enabled diverse applications with very different latency requirements. Existing LLM serving frameworks rely on siloed infrastructure with coarse-grained workload segregation -- interactive and batch -- leading to inefficient resource utilization and limited support for fine-grained Quality-of-Service (QoS) differentiation. This results in operational inefficiencies, over-provisioning and poor load management during traffic surges. We present Niyama, a novel QoS-driven inference serving system that enables efficient co-scheduling of diverse workloads on shared infrastructure. Niyama introduces fine-grained QoS classification allowing applications to specify precise latency requirements, and dynamically adapts scheduling decisions based on real-time system state. Leveraging the predictable execution characteristics of LLM inference, Niyama implements a dynamic chunking mechanism to improve overall throughput while maintaining strict QoS guarantees. Additionally, Niyama employs a hybrid prioritization policy that balances fairness and efficiency, and employs selective request relegation that enables graceful service degradation during overload conditions. Our evaluation demonstrates that Niyama increases serving capacity by 32% compared to current siloed deployments, while maintaining QoS guarantees. Notably, under extreme load, our system reduces SLO violations by an order of magnitude compared to current strategies.

Figures (12)

• Figure 1: Efficiency of Niyama under uniform load and transient overload conditions. (top left) Normalized GPUs needed to serve a fixed traffic load while meeting the QoS targets of requests divided equally among 3 QoS tiers. Across two datasets, Niyama improves efficiency by 12-32% compared to the State-of-the-art (SOTA) Sarathi-Serve sarathi2024 siloed deployments. (top right) Illustration of Niyama co-scheduling vs current siloed deployments. (bottom left) Bursty overload scenario. (bottom right) Niyama maintains low latency while SOTA scheduling succumbs to cascading deadline violations under such bursty loads.
• Figure 2: Comparison of traditional policies for multi-SLA scheduling. The graphs plot the latency and violations in the strictest QoS class. FCFS breaks down very quickly because urgent requests can be blocked by non-urgent ones. Deadline-aware policies like EDF are better than FCFS, but cannot gracefully degrade at high loads because of intense queue buildup. SJF/SRPF on the other hand can maintain QoS in the median case but violates SLOs of majority of long jobs even at a low load of 2.5 QPS. Niyama interpolates smoothly between SJF and EDF and minimizes violations across all load conditions.
• Figure 3: Overview of Niyama
• Figure 4: Performance characteristics as a function of chunk size, showing the throughput-latency tradeoff.
• Figure 5: Proactively relegating a small percentage of requests enormously helps is maintaining the quality of service for the median request in the system, which otherwise grows exponentially due to a cascade of violations.