MotionStream: Real-Time Video Generation with Interactive Motion Controls

TL;DR

The paper introduces MotionStream, a streaming autoregressive framework for motion-conditioned video generation that achieves sub-second latency and real-time interactivity by distilling a bidirectional motion-controlled teacher into a causal student. Central to the approach are attention sink mechanisms and rolling KV caches that enable stable, infinite-length generation, along with Self Forcing-style distribution matching distillation and joint text-motion guidance to preserve both trajectory fidelity and natural dynamics. A lightweight track-head and a Tiny VAE accelerate inference, delivering up to 29 FPS on a single GPU while maintaining state-of-the-art motion transfer and camera control performance. Extensive ablations illuminate the importance of track encoding, guidance fusion, and sparse attention settings for long-horizon stability. The work enables interactive use cases like motion transfer, drag-based controls, and real-time camera manipulation, representing a significant step toward open-domain, real-time video generation.

Abstract

Current motion-conditioned video generation methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present MotionStream, enabling sub-second latency with up to 29 FPS streaming generation on a single GPU. Our approach begins by augmenting a text-to-video model with motion control, which generates high-quality videos that adhere to the global text prompt and local motion guidance, but does not perform inference on the fly. As such, we distill this bidirectional teacher into a causal student through Self Forcing with Distribution Matching Distillation, enabling real-time streaming inference. Several key challenges arise when generating videos of long, potentially infinite time-horizons -- (1) bridging the domain gap from training on finite length and extrapolating to infinite horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without incurring growth in computational cost due to increasing context windows. A key to our approach is introducing carefully designed sliding-window causal attention, combined with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we properly simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Our models achieve state-of-the-art results in motion following and video quality while being two orders of magnitude faster, uniquely enabling infinite-length streaming. With MotionStream, users can paint trajectories, control cameras, or transfer motion, and see results unfold in real-time, delivering a truly interactive experience.

Figures (13)

• Figure 1: Prior motion-controlled video diffusion models typically operate offline to generate fixed-length sequences in parallel (top left). In contrast, our MotionStream enables streaming long-video generation from a single image with track control at interactive speed (bottom left). MotionStream can be applied to a variety of online downstream applications, such as real-time motion transfer, user drag operations, and 3D camera control (right).
• Figure 2: Model architecture and training pipeline. To build a teacher motion-controlled video model, we extract and randomly sample 2D tracks from the input video and encode them using a lightweight track head. The resulting track embeddings are combined with the input image, noisy video latents, and text embeddings as input to the diffusion transformer with bidirectional attention, which is then trained with a flow matching loss (top). We then distill a few-step causal diffusion model from the teacher through Self Forcing-style DMD distillation, integrating joint text-motion guidance into the objective, where autoregressive rollout with rolling KV cache and attention sink is applied during both training and inference (bottom).
• Figure 3: Visualization of self attention probability map. We visualize attention probability maps for bidirectional, full causal, and causal sliding window attentions. Several attention heads focus on the tokens corresponding to the initial frame throughout denoising generation.
• Figure 4: Quantitative ablation on guidance. We use Sora subset to ablate guidance strategies. Higher text guidance reduces overall metrics while motion guidance improves trajectory accuracy at the cost of visual quality (LPIPS).
• Figure 5: Qualitative ablation on guidance. Pure motion guidance produces rigid movements while text guidance enables natural motion and shape preservation even with imperfect tracks. Our Hybrid joint guidance balances these two.