VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understanding

TL;DR

VidLaDA introduces a bidirectional diffusion language model (DLM) for video understanding to address the limitations of autoregressive Video LLMs, enabling global spatiotemporal reasoning. It couples a robust vision encoder with a full bidirectional attention-based diffusion decoder and pairs it with MARS-Cache, a multimodal asynchronous refreshing strategy that exploits frame-wise locality, anchor tokens, and modality- and depth-aware caching to drastically accelerate inference. Empirical results show VidLaDA outperforms diffusion baselines and rivals state-of-the-art autoregressive models, while MARS-Cache provides over 12x throughput gains and up to 8–12x speedups under Chain-of-Thought inference without sacrificing accuracy. This work offers a practical, non-autoregressive paradigm for efficient long-context video understanding with strong reasoning capabilities, making diffusion-based Video LLMs more scalable and deployable.

Abstract

Standard Autoregressive Video LLMs inevitably suffer from causal masking biases that hinder global spatiotemporal modeling, leading to suboptimal understanding efficiency. We propose VidLaDA, a Video LLM based on Diffusion Language Model utilizing bidirectional attention to capture bidirectional dependencies. To further tackle the inference bottleneck of diffusion decoding on massive video tokens, we introduce MARS-Cache. This framework accelerates inference by combining asynchronous visual cache refreshing with frame-wise chunk attention, effectively pruning redundancy while preserving global connectivity via anchor tokens. Extensive experiments show VidLaDA outperforms diffusion baselines and rivals state-of-the-art autoregressive models (e.g., Qwen2.5-VL and LLaVA-Video), with MARS-Cache delivering over 12x speedup without compromising reasoning accuracy. Code and checkpoints are open-sourced at https://github.com/ziHoHe/VidLaDA.

Figures (10)

• Figure 1: The overall architecture of VidLaDA. Input video frames are encoded and spatially pooled (via $2\times2$ downsampling) before being unrolled into a sequence of Spatiotemporal Visual Tokens $V$. These tokens, combined with the text prompt $P$ and the noised answer $X_t$, are processed by the Diffusion Language Model. Unlike autoregressive models, VidLaDA utilizes full bidirectional attention. This design enables global, unconstrained interactions both within and across visual and textual modalities, ultimately facilitating the parallel prediction of the target answer $X_0$.
• Figure 2: Comparison of Spatiotemporal Robustness.(a) Performance vs. spatial location of high-norm tokens. DLM-Based VLM remains invariant, whereas AR degrades when salient features shift from the start. (b) Performance vs. temporal location of the key event. DLM-Based VLM demonstrates stability across the timeline, while AR baselines show significant volatility. (c) DLM-Based VLM maintains high accuracy with fewer frames, demonstrating superior aggregation of sparse evidence compared to AR models.
• Figure 3: Visualization of Attention Patterns. We display the attention score matrices across different denoising steps and layers. The heatmaps reveal two distinct structural properties utilized by our MARS-Cache design: (1) Chunk-wise Locality, visible as diagonal blocks where tokens primarily attend to their temporal neighbors, and (2) Global Anchor Tokens, manifested as prominent vertical bands where specific tokens consistently attract global attention from the entire sequence, regardless of the diffusion step or network depth.
• Figure 4: Modality-Dependent State Evolution. We visualize the cosine similarity matrix of hidden states across different inference steps. Left: Visual tokens exhibit high stability, indicating low drift during the decoding process. Right: Textual tokens show lower similarity between steps, indicating high volatility.
• Figure 5: Depth-Dependent Hidden State Drift. The heatmap illustrates the magnitude of hidden state drift (measured as 1$-$Cosine Similarity) across network layers (Y-axis) and inference steps (X-axis). Shallow layers remain stable with minimal drift, whereas deep layers exhibit significant volatility.

Theorems & Definitions (4)

• Proposition 3.1
• proof
• Proposition 3.2
• proof