DISK: Dynamic Inference SKipping for World Models

TL;DR

The paper tackles the high latency of autoregressive diffusion-based world models used for long-horizon planning in driving. It introduces DISK, a training-free adaptive inference module that coordinates two diffusion transformers (VisDiT for video and TrajDiT for ego-trajectory) with per-step compute/skip decisions over the diffusion steps $K$, guided by a cross-modal safety gate. DISK preserves motion-appearance consistency and long-horizon stability by propagating lightweight controller statistics through the autoregressive loop. On NuPlan and NuScenes with an NVIDIA L40S, it achieves ~2x speedup on the trajectory branch and ~1.6x on the video branch while maintaining $L2$ planning error, $FID$/$FVD$, and NAVSIM $PDMS$, demonstrating practical real-time performance without retraining.

Abstract

We present DISK, a training-free adaptive inference method for autoregressive world models. DISK coordinates two coupled diffusion transformers for video and ego-trajectory via dual-branch controllers with cross-modal skip decisions, preserving motion-appearance consistency without retraining. We extend higher-order latent-difference skip testing to the autoregressive chain-of-forward regime and propagate controller statistics through rollout loops for long-horizon stability. When integrated into closed-loop driving rollouts on 1500 NuPlan and NuScenes samples using an NVIDIA L40S GPU, DISK achieves 2x speedup on trajectory diffusion and 1.6x speedup on video diffusion while maintaining L2 planning error, visual quality (FID/FVD), and NAVSIM PDMS scores, demonstrating practical long-horizon video-and-trajectory prediction at substantially reduced cost.

Figures (4)

• Figure 1: DISK Overview. During diffusion sampling, dual-branch controllers decide per-step whether to compute (filled) or skip (dashed) each denoising evaluation. A safety gate synchronizes branches when the trajectory encounters complex maneuvers, ensuring motion-appearance consistency. DISK achieves $\sim$2$\times$ speedup on trajectory diffusion and $\sim$1.6$\times$ on video diffusion while preserving output quality.
• Figure 2: Qualitative Video Comparison. Video frames from 11-second rollouts show visually indistinguishable quality between baseline (Epona) and DISK despite $\sim$1.6$\times$ speedup in vision diffusion. DISK maintains high fidelity and temporal consistency across both NuPlan and NuScenes datasets.
• Figure 3: Overview of DISK inference. The Trajectory Controller influences the Vision branch via a unidirectional Safety Gate, ensuring visual consistency during complex maneuvers. The Vision Controller operates independently but is overridden if the Safety Gate triggers. Statistics from both branches seed the next autoregressive step.
• Figure 4: Adaptive Skipping Pattern. Example skip decisions across 11 diffusion steps for vision and trajectory branches. Filled circles indicate compute steps, crosses indicate skips. The safety gate triggers (dotted lines) force vision compute when trajectory encounters complex maneuvers.