Evaluating Kubernetes Performance for GenAI Inference: From Automatic Speech Recognition to LLM Summarization

TL;DR

The paper tackles the challenge of efficiently deploying GenAI inference workloads on Kubernetes by evaluating an end-to-end pipeline (ASR via Whisper followed by LLM-based summarization) using three Kubernetes-native components: Kueue for batch scheduling, DAS for dynamic GPU slicing, and GAIE for distributed inference routing via llm-d. It demonstrates how these components can be composed to achieve improved makespan, reduced completion times, and lower latency under high concurrency, validated on a realistic Earnings22 dataset with Whisper and Qwen-based models. The key contributions are the end-to-end evaluation of a full GenAI inference stack on Kubernetes, quantified performance gains (e.g., reduced makespan, faster time-to-first-token, and better throughput) and a clear design direction for production-grade AI orchestration on cloud-native infrastructure. The work shows that Kubernetes can serve as a unified substrate for demanding GenAI workloads when extended with domain-specific scheduling, resource slicing, and model-aware routing, with practical impact for scalable, multi-tenant AI deployments.

Abstract

As Generative AI (GenAI), particularly inference, rapidly emerges as a dominant workload category, the Kubernetes ecosystem is proactively evolving to natively support its unique demands. This industry paper demonstrates how emerging Kubernetes-native projects can be combined to deliver the benefits of container orchestration, such as scalability and resource efficiency, to complex AI workflows. We implement and evaluate an illustrative, multi-stage use case consisting of automatic speech recognition and summarization. First, we address batch inference by using Kueue to manage jobs that transcribe audio files with Whisper models and Dynamic Accelerator Slicer (DAS) to increase parallel job execution. Second, we address a discrete online inference scenario by feeding the transcripts to a Large Language Model for summarization hosted using llm-d, a novel solution utilizing the recent developments around the Kubernetes Gateway API Inference Extension (GAIE) for optimized routing of inference requests. Our findings illustrate that these complementary components (Kueue, DAS, and GAIE) form a cohesive, high-performance platform, proving Kubernetes' capability to serve as a unified foundation for demanding GenAI workloads: Kueue reduced total makespan by up to 15%; DAS shortened mean job completion time by 36%; and GAIE improved Time to First Token by 82\%.

Figures (7)

• Figure 1: Overview of the technology stack.
• Figure 2: Kueue working mechanism.
• Figure 3: Kueue setup comparison for Whisper transcription experiments.
• Figure 4: Overview of the llm-d setup.
• Figure 5: Comparison of the overhead introduced by different scenarios. Error bars indicate 95% confidence interval.