MATRIX: Mask Track Alignment for Interaction-aware Video Generation

TL;DR

This work addresses the challenge of modeling multi-instance interactions in video diffusion transformers by introducing MATRIX-11K, a dataset with interaction-aware captions and per-instance mask tracks, and InterGenEval for interaction-focused evaluation. Through a systematic analysis of 3D full attention, the authors identify interaction-dominant layers where semantic grounding (via video-to-text) and semantic propagation (via video-to-video) concentrate. They then propose MATRIX, a lightweight regularization that aligns attention in those layers with ground-truth mask tracks using Semantic Grounding Alignment (SGA) and Semantic Propagation Alignment (SPA) losses, guided by a lightweight causal decoder and LoRA-based fine-tuning. Empirical results on InterGenEval show that MATRIX improves interaction fidelity and semantic alignment while reducing drift and hallucination, with ablations confirming the importance of layer selection and the complementary roles of SGA and SPA. The dataset and framework offer a practical path toward more reliable, interaction-aware video generation in diffusion-era models.

Abstract

Video DiTs have advanced video generation, yet they still struggle to model multi-instance or subject-object interactions. This raises a key question: How do these models internally represent interactions? To answer this, we curate MATRIX-11K, a video dataset with interaction-aware captions and multi-instance mask tracks. Using this dataset, we conduct a systematic analysis that formalizes two perspectives of video DiTs: semantic grounding, via video-to-text attention, which evaluates whether noun and verb tokens capture instances and their relations; and semantic propagation, via video-to-video attention, which assesses whether instance bindings persist across frames. We find both effects concentrate in a small subset of interaction-dominant layers. Motivated by this, we introduce MATRIX, a simple and effective regularization that aligns attention in specific layers of video DiTs with multi-instance mask tracks from the MATRIX-11K dataset, enhancing both grounding and propagation. We further propose InterGenEval, an evaluation protocol for interaction-aware video generation. In experiments, MATRIX improves both interaction fidelity and semantic alignment while reducing drift and hallucination. Extensive ablations validate our design choices. Codes and weights will be released.

Figures (33)

• Figure 1: Teaser: We reveal how video diffusion transformers (DiTs) represent multi-instance or subject-object interactions during video generation. Building on this, our MATRIX framework further enhances the interaction-awareness of video DiTs via the proposed Semantic Grounding Alignment (SGA, $\mathcal{L}_\mathrm{SGA}$) and Semantic Propagation Alignment (SPA, $\mathcal{L}_\mathrm{SPA}$) losses.
• Figure 2: Failure cases of existing video DiTs: (a) semantic grounding failures, where subjects, objects, or their verb relations are mismatched, and (b) semantic propagation failures, where bindings break over time, leading to hallucinations or duplications. Overlays indicate the intended instances.
• Figure 3: Attention maps per token type. Noun tokens (subject, object) align with their respective regions (e.g., layer 11); verb tokens aligns with the union of subject–object regions (e.g., layer 7).
• Figure 4: Our dataset curation pipeline. An LLM identifies interaction triplets, filters them using Dynamism and Contactness, and extracts per-ID appearance descriptions (Sec. \ref{['dataset:caption_processing']}). A VLM then verifies candidate to select an anchor frame, from which SAM2 propagates masks to produce instance mask tracks $M_k$. We drop instances and related interactions that fail verification or propagation (Sec. \ref{['dataset:mask_track_extraction']})
• Figure 5: Illustration of full 3D attention in video DiTs.