LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference

TL;DR

LatencyPrism tackles the challenge of guaranteeing SLA-compliant latency for production LLM inference amid dynamic, multi-tenant, heterogeneous hardware. It introduces a non-intrusive, cross-stack observability framework built on perception, comprehension, and adaptation layers, enabling online anomaly detection with on-demand deep tracing. The system achieves negligible overhead (less than 0.5% CPU) and high detection fidelity (F1 ≈ 0.985), supported by a physically informed modeling approach and cycle-based root-cause localization across CPUs, GPUs, and interconnects. This work offers a practical path to reliable, low-latency LLM services by bridging high-level application semantics with low-level hardware signals in real time.

Abstract

LLM inference latency critically determines user experience and operational costs, directly impacting throughput under SLO constraints. Even brief latency spikes degrade service quality despite acceptable average performance. However, distributed inference environments featuring diverse software frameworks and XPU architectures combined with dynamic workloads make latency analysis challenging. Constrained by intrusive designs that necessitate service restarts or even suspension, and by hardware-bound implementations that fail to adapt to heterogeneous inference environments, existing AI profiling methods are often inadequate for real-time production analysis. We present LatencyPrism, the first zero-intrusion multi-platform latency sculpting system. It aims to break down the inference latency across pipeline, proactively alert on inference latency anomalies, and guarantee adherence to SLOs, all without requiring code modifications or service restarts. LatencyPrism has been deployed across thousands of XPUs for over six months. It enables low-overhead real-time monitoring at batch level with alerts triggered in milliseconds. This approach distinguishes between workload-driven latency variations and anomalies indicating underlying issues with an F1-score of 0.98. We also conduct extensive experiments and investigations into root cause analysis to demonstrate LatencyPrism's capability.

Figures (6)

• Figure 1: Architecture Overview of LatencyPrism. The system consists of three layers: (1) The Perception Layer collects non-intrusive runtime events and telemetry; (2) The Comprehension Layer aligns cross-stack events and constructs workload-aware baselines; (3) The Adaptation Layer monitors latency residuals, triggering alerts providing deep-dive contexts when anomalies detected.
• Figure 2: Distribution of Prefill stage duration in a production environment. Even with fixed input lengths, execution time fluctuates drastically with a long tail, primarily due to KV Cache hit rate variations.
• Figure 3: Detailed implementation workflow of LatencyPrism. (a) Maps the heterogeneous inference environment (L1-L5) to non-intrusive collectors via Shared Memory. (b) Illustrates the pipeline from trace alignment to GBDT-based anomaly detection. The system flows from left (collection) to right (decision).
• Figure 4: Trace view demonstrating cross-stack semantic alignment. LatencyPrism aligns CUDA Runtime API calls with underlying GPU kernel executions (e.g., AddPaddingkernel) on a unified timeline.
• Figure 5: Model Convergence Analysis. The model achieves rapid stabilization across three distinct scenarios: (a) unseen workloads in SGLang; (b) raw production noise without denoising; and (c) a cross-stack environment (vLLM, DeepSeek-70B), demonstrating robust generalization capabilities.