VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer

TL;DR

This work addresses the need for controllable animated VFX generation by presenting Open-VFX, a high-quality dataset of 675 VFX videos across 15 effects with textual prompts, instance masks, and start-end timestamps, and VFX Creator, a Video Diffusion Transformer-based framework. The model employs spatial and temporal LoRA adapters, including a plug-and-play mask conditioning module and two temporal integration strategies to achieve fine-grained control over effect timing and motion rhythm with minimal training data. Comprehensive evaluations show state-of-the-art performance in both spatial and temporal controllability, along with a novel metric for temporal precision and a user study confirming semantic alignment. Overall, this work advances practical, data-efficient VFX generation, enabling broader accessibility and faster production workflows in film and media.

Abstract

Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences. While recent advances in generative artificial intelligence have driven progress in generic image and video synthesis, the domain of controllable VFX generation remains relatively underexplored. In this work, we propose a novel paradigm for animated VFX generation as image animation, where dynamic effects are generated from user-friendly textual descriptions and static reference images. Our work makes two primary contributions: (i) Open-VFX, the first high-quality VFX video dataset spanning 15 diverse effect categories, annotated with textual descriptions, instance segmentation masks for spatial conditioning, and start-end timestamps for temporal control. (ii) VFX Creator, a simple yet effective controllable VFX generation framework based on a Video Diffusion Transformer. The model incorporates a spatial and temporal controllable LoRA adapter, requiring minimal training videos. Specifically, a plug-and-play mask control module enables instance-level spatial manipulation, while tokenized start-end motion timestamps embedded in the diffusion process, alongside the text encoder, allow precise temporal control over effect timing and pace. Extensive experiments on the Open-VFX test set demonstrate the superiority of the proposed system in generating realistic and dynamic effects, achieving state-of-the-art performance and generalization ability in both spatial and temporal controllability. Furthermore, we introduce a specialized metric to evaluate the precision of temporal control. By bridging traditional VFX techniques with generative approaches, VFX Creator unlocks new possibilities for efficient and high-quality video effect generation, making advanced VFX accessible to a broader audience.

Figures (9)

• Figure 1: Overview of our proposed Open-VFX Dataset. (a) demonstrates diverse input inference images in the dataset, including humans, animals, objects, and various scenes across single and multiple components. (b) shows the text descriptions of the proposed 15 VFXs, and (c) presents an example (Explode it) VFX.
• Figure 2: More examples of our Open-VFX dataset, including 10 VFXs and diverse reference images.
• Figure 3: Pipeline of VFX Creator. We introduce two novel modules: (a) Spatial Controlled LoRA Adapter. This module integrates a mask-conditioned ControlNet with LoRA, injecting mask sequences into the model to enable instance-level spatial manipulation. (b) Temporal Controlled LoRA Adapter. We explore two strategies for incorporating temporal control: module I involves tokenizing start-end motion timestamps and embedding them into the diffusion process alongside the text space, while module II integrates temporal mask with timestep embeddings to guide the diffusion process.
• Figure 4: Qualitative comparisons of VFX video generation on two different visual effects between our method, DynamiCrafter, LTX-Video, CogVideoX-5B, and Pika.
• Figure 5: Qualitative results of spatial controllable VFX video generation of our method on two different visual effects. Users can precisely specify the animated instance by clicking points or dropping boxes to obtain the mask.