DreaMontage: Arbitrary Frame-Guided One-Shot Video Generation

TL;DR

DreaMontage tackles the challenge of generating seamless long-form one-shot videos guided by arbitrary frames, overcoming limitations of naive clip concatenation. It introduces an intermediate-conditioning mechanism within a Diffusion Transformer backbone, a Visual Expression SFT pipeline with curated data, a Tailored DPO objective to reduce abrupt cuts and implausible motion, and a Segment-wise Auto-Regressive inference strategy for memory-efficient long sequences. The method is validated through extensive experiments showing improved temporal coherence, motion realism, and fidelity against state-of-the-art baselines, across both multi-keyframe and first-last conditioning settings. The approach enables practical, high-quality cinematic storytelling from mixed inputs and supports scalable generation of long, cohesive videos for creative and industrial workflows.

Abstract

The "one-shot" technique represents a distinct and sophisticated aesthetic in filmmaking. However, its practical realization is often hindered by prohibitive costs and complex real-world constraints. Although emerging video generation models offer a virtual alternative, existing approaches typically rely on naive clip concatenation, which frequently fails to maintain visual smoothness and temporal coherence. In this paper, we introduce DreaMontage, a comprehensive framework designed for arbitrary frame-guided generation, capable of synthesizing seamless, expressive, and long-duration one-shot videos from diverse user-provided inputs. To achieve this, we address the challenge through three primary dimensions. (i) We integrate a lightweight intermediate-conditioning mechanism into the DiT architecture. By employing an Adaptive Tuning strategy that effectively leverages base training data, we unlock robust arbitrary-frame control capabilities. (ii) To enhance visual fidelity and cinematic expressiveness, we curate a high-quality dataset and implement a Visual Expression SFT stage. In addressing critical issues such as subject motion rationality and transition smoothness, we apply a Tailored DPO scheme, which significantly improves the success rate and usability of the generated content. (iii) To facilitate the production of extended sequences, we design a Segment-wise Auto-Regressive (SAR) inference strategy that operates in a memory-efficient manner. Extensive experiments demonstrate that our approach achieves visually striking and seamlessly coherent one-shot effects while maintaining computational efficiency, empowering users to transform fragmented visual materials into vivid, cohesive one-shot cinematic experiences.

Figures (7)

• Figure 1: DreaMontage: Flexible Dreams, Seamless Montage. Our model generates one-shot, long-form videos guided by arbitrary keyframes or video clips anchored at precise temporal locations.
• Figure 2: Overview of the DreaMontage. The left panel illustrates the multi-stage training pipeline, progressing from the Adaptive Tuning to the Visual Expression SFT and Tailored DPO. The right panel depicts the inference pipeline, where reference (condition) images/videos and rephrased prompts guide the generation process, supporting auto-regressive long-video generation.
• Figure 3: The Interm-Cond Adaptation strategy. (a) Due to the Causalty VAE's temporal downsampling, an intermediate latent aggregates information from multiple frames, making it an imprecise condition for a specific timestamp. (b) To resolve this, we align the training distribution with inference. Each single condition frame (or the initial frame of a condition video) is re-encoded while the subsequent frames of the condition video are re-sampled from the latent distribution.
• Figure 4: The Shared-RoPE strategy for the super-resolution model. In addition to channel-wise concatenation, we introduce a sequence-wise conditioning mechanism to eliminate artifacts. Condition frames are appended to the tail of the sequence while share the same RoPE value as the target frames they guide (e.g., $C_i$ shares the RoPE of $t_1$). In the case of video condition, this strategy is only applied to the first frame.
• Figure 5: Illustration of the Tailored DPO. To eliminate specific generation artifacts, we construct preference pairs via two distinct pipelines: Pipeline A addresses abrupt cuts by leveraging a trained VLM discriminator to automatically select positive/negative samples, whereas Pipeline B targets subject motion rationality through human-annotated screening of challenging cases. These pairs subsequently drive the DPO training to optimize the policy $\pi_\theta$ against the reference model $\pi_{\text{ref}}$, ensuring smoother transitions and physically plausible motions.