VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models

TL;DR

<3-5 sentence high-level summary> VideoMaker targets zero-shot customized video generation by exploiting the inherent feature extraction and injection forces of pre-trained Video Diffusion Models (VDMs). It eliminates external subject extractors and injectors by feeding a noise-free reference image into the VDM at t=0 for fine-grained feature extraction and by using the model's native spatial self-attention to inject subject cues into per-frame latent representations. A Guidance Information Recognition Loss facilitates stable discrimination between reference information and generated content, enabling reliable customization while preserving video diversity. Extensive experiments on customized human and object generation show improved subject fidelity and text alignment over strong baselines, with ablations highlighting the contributions of the injection mechanism, loss design, and preprocessing. The approach offers a practical, training-efficient path toward realistic, personalized video generation without expensive retraining or auxiliary modules.</br>

Abstract

Zero-shot customized video generation has gained significant attention due to its substantial application potential. Existing methods rely on additional models to extract and inject reference subject features, assuming that the Video Diffusion Model (VDM) alone is insufficient for zero-shot customized video generation. However, these methods often struggle to maintain consistent subject appearance due to suboptimal feature extraction and injection techniques. In this paper, we reveal that VDM inherently possesses the force to extract and inject subject features. Departing from previous heuristic approaches, we introduce a novel framework that leverages VDM's inherent force to enable high-quality zero-shot customized video generation. Specifically, for feature extraction, we directly input reference images into VDM and use its intrinsic feature extraction process, which not only provides fine-grained features but also significantly aligns with VDM's pre-trained knowledge. For feature injection, we devise an innovative bidirectional interaction between subject features and generated content through spatial self-attention within VDM, ensuring that VDM has better subject fidelity while maintaining the diversity of the generated video. Experiments on both customized human and object video generation validate the effectiveness of our framework.

Figures (13)

• Figure 1: The overview of the celebrity dataset we use to test customized human video generation.
• Figure 2: Compared with the existing zero-shot customized generation framework. Our framework does not require any additional modules to extract or inject subject features. It only needs simple concatenation of the reference image and generated video, and VDM's inherent force is used to generate custom video.
• Figure 2: The overview of the dataset we use to test customized object video generation.
• Figure 3: Overall pipeline of VideoMaker. We directly input the reference image into VDM and use VDM's modules for fine-grained feature extraction. We modified the computation of spatial self-attention to enable feature injection. Additionally, to distinguish between reference features and generated content, we designed the Guidance Information Recognition Loss to optimize the training strategy.
• Figure 3: The overview of the non-celebrity dataset we used for testing customized human video generation.