Early Failure Detection and Intervention in Video Diffusion Models

Abstract

Text-to-video (T2V) diffusion models have rapidly advanced, yet generations still occasionally fail in practice, such as low text-video alignment or low perceptual quality. Since diffusion sampling is non-deterministic, it is difficult to know during inference whether a generation will succeed or fail, incurring high computational cost due to trial-and-error regeneration. To address this, we propose an early failure detection and diagnostic intervention pipeline for latent T2V diffusion models. For detection, we design a Real-time Inspection (RI) module that converts latents into intermediate video previews, enabling the use of established text-video alignment scorers for inspection in the RGB space. The RI module completes the conversion and inspection process in just 39.2ms. This is highly efficient considering that CogVideoX-5B requires 4.3s per denoising step when generating a 480p, 49-frame video on an NVIDIA A100 GPU. Subsequently, we trigger a hierarchical and early-exit intervention pipeline only when failure is predicted. Experiments on CogVideoX-5B and Wan2.1-1.3B demonstrate consistency gains on VBench with up to 2.64 times less time overhead compared to post-hoc regeneration. Our method also generalizes to a higher-capacity setting, remaining effective on Wan2.1-14B with 720p resolution and 81-frame generation. Furthermore, our pipeline is plug-and-play and orthogonal to existing techniques, showing seamless compatibility with prompt refinement and sampling guidance methods. We also provide evidence that failure signals emerge early in the denoising process and are detectable within intermediate video previews using standard vision-language evaluators.

Figures (24)

• Figure 1: Real-time inspection enables early failure detection in latent video diffusion. By decoding intermediate latents at every denoising step, our method identifies failing trajectories early and selectively intervenes. Compared to post-hoc regeneration, we achieve comparable VBench scores with 2.64× lower time overhead.
• Figure 2: Overall early failure detection and intervention pipeline. The Real-time Inspection (RI) module converts predicted clean latents into RGB video previews and evaluates their semantic alignment with the input prompt (19.7 ms + 19.5 ms). Intervention is triggered based on the semantic alignment score.
• Figure 3: Real-time Latent-to-RGB (L2R) converter. (a) Our L2R architecture maps latents to an RGB video using causal 3D convolutional blocks, spatio-temporal learned upsampling, and a projection layer. It runs in 19.7 ms with only 0.059 M parameters. (b) The reconstruction result from the L2R converter shows reasonable visual quality compared to the result produced by the original decoder.
• Figure 4: Diagnostic intervention framework. If the current generation is predicted to succeed, Trial 0 continues to generate video without intervention. When failure is predicted, the next Trials are applied, escalating from inexpensive to more costly fixes. Trial 1 performs a single-frame probe and injects its latent; Trial 2 refines the prompt via a VLM and restarts generation. If Trial 2 also fails, Trial 1 is attempted once more.
• Figure 5: Efficiency of selective early intervention. (a) Overall time overhead relative to base generation. Ours incurs only +16.6% additional overhead, substantially lower than the regeneration of failure samples (+43.7%). (b) Per-sample cost increases with deeper trials, yet remains cheaper than regeneration. Since deeper trials are triggered for only a subset of samples (red curve), the overall time overhead stays modest.