STANCE: Motion Coherent Video Generation Via Sparse-to-Dense Anchored Encoding

TL;DR

STANCE tackles the problem of preserving coherent object motion in video generation by converting sparse, user-provided hints into dense, pixel-aligned Instance Cues and by enriching token space with Dense RoPE to retain spatial identities over time. It further stabilizes training and geometry through joint RGB plus auxiliary (depth or segmentation) supervision that attends to the same cues. Evaluations on a 200k-clip Kubric-derived dataset and real-world tests show improved motion coherence (Physics IQ) and reduced drift compared with baselines, while maintaining high visual quality. The approach enables practical, script-free control of dynamics across simple and realistic scenes, advancing physics-aware video synthesis for editing and robotics applications.

Abstract

Video generation has recently made striking visual progress, but maintaining coherent object motion and interactions remains difficult. We trace two practical bottlenecks: (i) human-provided motion hints (e.g., small 2D maps) often collapse to too few effective tokens after encoding, weakening guidance; and (ii) optimizing for appearance and motion in a single head can favor texture over temporal consistency. We present STANCE, an image-to-video framework that addresses both issues with two simple components. First, we introduce Instance Cues -- a pixel-aligned control signal that turns sparse, user-editable hints into a dense 2.5D (camera-relative) motion field by averaging per-instance flow and augmenting with monocular depth over the instance mask. This reduces depth ambiguity compared to 2D arrow inputs while remaining easy to use. Second, we preserve the salience of these cues in token space with Dense RoPE, which tags a small set of motion tokens (anchored on the first frame) with spatial-addressable rotary embeddings. Paired with joint RGB \(+\) auxiliary-map prediction (segmentation or depth), our model anchors structure while RGB handles appearance, stabilizing optimization and improving temporal coherence without requiring per-frame trajectory scripts.

Figures (6)

• Figure 1: Videos generated by STANCE. User input: one keyframe, coarse 2D arrows, per-instance mass, and a scalar depth delta $\Delta z$. Controls yield physically meaningful edits while preserving appearance: increasing mass can reverse collision outcomes, larger speeds produce longer travel and earlier contact, and rotating the arrow reorients trajectories and shifts contact points; $\Delta z$ disambiguates out-of-plane intent under camera motion. Examples span both simple collision setups and realistic scenes, including gentle pushes that dislodge or trigger collisions.
• Figure 2: Pipeline of STANCE. Our method is organized as follows: (1) Left: we extend the input of DiT to include new alpha tokens, and use a train-able MLP to tokenize instance cues. (2) Right: The modality embeddings are added to the auxiliary tokens, and the instance cue tokens are paired with Dense RoPE.
• Figure 3: Left:2D Map vs. Dense RoPE. When the 2D control map is downsampled, many tokens inside the window become zeros, yielding a sparse signal that weakens control. Dense RoPE performs non-zero token extraction over the target region (colored), preserves/assigns positional embeddings ($e_1,\dots,e_n$), and feeds a compact, dense sequence to the model—resulting in stronger, spatially focused control. Right:Depth control. The upward black arrow is the user-drawn 2D arrow; by manipulating a scalar depth delta ($\Delta$ depth on the horizontal axis, $[-1,+1]$), the user specifies out-of-plane motion: $\Delta>0$ (red) points into the screen (away from the camera), while $\Delta<0$ (blue) points out of the screen (toward the camera).
• Figure 4: Applications.Left: Provide with the first frame, we alter the magnitude of the velocity and mass for the synthetic objects. Right: For realistic objects, the mass is fixed, thus we altered the direction and magnitude of velocity only. The video clips for all shown cases can be located in the HTML files.
• Figure 5: Qualitative Comparisons. We compare against recent baselines. All videos are temporally aligned (trimmed or padded to a fixed duration) and spatially normalized (resized for visualization). STANCE attains superior visual fidelity while maintaining strong physical coherence. The video clips for all shown cases can be located in the HTML files.