Making MoE-based LLM Inference Resilient with Tarragon

TL;DR

Tarragon addresses the fragility of MoE-based LLM inference by partitioning failure domains into Attention Workers and Expert Workers using a reconfigurable AW-EW datapath (REFE) and an Expert Routing Table (ERT). It couples this with self-healing mechanisms, shadow experts, and asynchronous KV-cache checkpointing to confine failures to local domains and avoid global stalls, achieving end-to-end failover reductions of up to 160–213x while preserving steady-state throughput. The system introduces per-request KV-cache restoration and background provisioning to minimize recovery cost, with KV-cache checkpointing incurring negligible overhead. Empirical evaluation on Mixtral-8x7B demonstrates dramatic reductions in stall times under failures and near-parity with state-of-the-art decoupled MoE serving under normal operation, highlighting Tarragon’s practical impact for production-scale LLM inference.

Abstract

Mixture-of-Experts (MoE) models are increasingly used to serve LLMs at scale, but failures become common as deployment scale grows. Existing systems exhibit poor failure resilience: even a single worker failure triggers a coarse-grained, service-wide restart, discarding accumulated progress and halting the entire inference pipeline during recovery--an approach clearly ill-suited for latency-sensitive, LLM services. We present Tarragon, a resilient MoE inference framework that confines the failures impact to individual workers while allowing the rest of the pipeline to continue making forward progress. Tarragon exploits the natural separation between the attention and expert computation in MoE-based transformers, treating attention workers (AWs) and expert workers (EWs) as distinct failure domains. Tarragon introduces a reconfigurable datapath to mask failures by rerouting requests to healthy workers. On top of this datapath, Tarragon implements a self-healing mechanism that relaxes the tightly synchronized execution of existing MoE frameworks. For stateful AWs, Tarragon performs asynchronous, incremental KV cache checkpointing with per-request restoration, and for stateless EWs, it leverages residual GPU memory to deploy shadow experts. These together keep recovery cost and recomputation overhead extremely low. Our evaluation shows that, compared to state-of-the-art MegaScale-Infer, Tarragon reduces failure-induced stalls by 160-213x (from ~64 s down to 0.3-0.4 s) while preserving performance when no failures occur.

Figures (15)

• Figure 1: (a) MoE-based transformer layer and LLM inference pipeline: example shows top-$2$ experts selected; (b) decoupled attention-expert deployment.
• Figure 2: Example of layer-wise synchronized MoE inference. Here we show two data-parallel AWs, and two EWs, each hosting three expert FFNs (E1-E6), the same as Fig. \ref{['fig:moe-overview']}(b).
• Figure 3: Coarse‐grained failure recovery under different deployment modes. Numbers inside the "decoding" boxes denote the transformer layer currently being executed. For the decoupled deployment, we analyze a "best-case" recovery scenario, where a single worker failure (AW or EW) results only in the failed worker restarting. However, some existing decoupled systems still restart all workers on failure, thus effectively degenerating to the monolithic case.
• Figure 4: (a–c) Inference stall time and (d–f) re-execution cost under a single worker failure. MO: monolithic worker.
• Figure 5: Overview of Tarragon.