Towards Resiliency in Large Language Model Serving with KevlarFlow

TL;DR

This paper addresses the fragility of large language model (LLM) serving in hyperscale clusters, where hardware faults cause prolonged outages. It introduces Kevlar-Flow, a fault-tolerant architecture that decouples model-parallel initialization, performs dynamic traffic rerouting, and continuously replicates KV cache to mask failures and keep requests flowing. Key findings show a 20x reduction in mean-time-to-recovery (MTTR) and substantial latency improvements under partial failures (average latency up to 3.1x; TTFT up to 378.9x) with negligible overhead during normal operation. The approach enables near-continuous service in the face of node failures, offering a practical, scalable path toward “always-on” LLM serving for large-scale AI systems.

Abstract

Large Language Model (LLM) serving systems remain fundamentally fragile, where frequent hardware faults in hyperscale clusters trigger disproportionate service outages in the software stack. Current recovery mechanisms are prohibitively slow, often requiring up to 10 minutes to reinitialize resources and reload massive model weights. We introduce KevlarFlow, a fault tolerant serving architecture designed to bridge the gap between hardware unreliability and service availability. KevlarFlow leverages 1) decoupled model parallelism initialization, 2) dynamic traffic rerouting, and 3) background KV cache replication to maintain high throughput during partial failures. Our evaluation demonstrates that KevlarFlow reduces mean-time-to-recovery (MTTR) by 20x and, under failure conditions, improves average latency by 3.1x, 99th percentile (p99) latency by 2.8x, average time-to-first-token (TTFT) by 378.9x, and p99 TTFT by 574.6x with negligible runtime overhead in comparison to state-of-the-art LLM serving systems.

Figures (9)

• Figure 1: Rolling average and p99 TTFT comparison between a typical LLM serving framework (baseline) and Kevlar-Flow under 2 requests per second (RPS). The red line indicates the time when one node fails. Y-axis is in log scale.
• Figure 2: Design of Kevlar-Flow. The example shows a 4-stage pipeline parallelism serving (i.e., the model weights are split to four model executors). The load balancing group has four active model instances. A red "X" indicate a failed node.
• Figure 3: Baseline latency performance of TensorRT-LLM. The performance of 8-node cluster is marked with blue dots, and that of the 16-node cluster is marked with orange triangles. Figures on the left are for the average latency, and figures on the right are for the p99 latency.
• Figure 4: Baseline TTFT of TensorRT-LLM. The performance of 8-node cluster is marked with blue dots, and that of the 16-node cluster is marked with orange triangles. Figures on the left are for the average TTFT, and figures on the right are for the p99 TTFT.
• Figure 5: Comparison of Kevlar-Flow and baseline under three node failure scenarios. Performance of the standard fault behavior is plotted using blue dots, Kevlar-Flow's performance is plotted in orange triangles. The improvement of Kevlar-Flow over the standard fault behavior is plotted in green dotted lines. The highest improvement is annotated in each figure.