Tuning-free Visual Effect Transfer across Videos

TL;DR

RefVFX presents a tuning-free framework for transferring complex temporal visual effects from a reference video to a target video or image using a diffusion backbone conditioned on the reference, input, and text. It leverages a large, multimodal dataset of over 120K triplets spanning ~1,700 temporal effects, built from LoRA-based I2V data, scalable V2V pipelines, and synthetic code-based effects. Through a multi-source conditioning architecture and feed-forward inference, RefVFX achieves coherent temporal dynamics and strong reference adherence while preserving input content, outperforming prompt-only and static-reference baselines in qualitative and human preference studies. The approach enables robust, generalizable temporal effect transfer with practical implications for editing and post-production, albeit with higher inference latency due to dual conditioning during generation.

Abstract

We present RefVFX, a new framework that transfers complex temporal effects from a reference video onto a target video or image in a feed-forward manner. While existing methods excel at prompt-based or keyframe-conditioned editing, they struggle with dynamic temporal effects such as dynamic lighting changes or character transformations, which are difficult to describe via text or static conditions. Transferring a video effect is challenging, as the model must integrate the new temporal dynamics with the input video's existing motion and appearance. % To address this, we introduce a large-scale dataset of triplets, where each triplet consists of a reference effect video, an input image or video, and a corresponding output video depicting the transferred effect. Creating this data is non-trivial, especially the video-to-video effect triplets, which do not exist naturally. To generate these, we propose a scalable automated pipeline that creates high-quality paired videos designed to preserve the input's motion and structure while transforming it based on some fixed, repeatable effect. We then augment this data with image-to-video effects derived from LoRA adapters and code-based temporal effects generated through programmatic composition. Building on our new dataset, we train our reference-conditioned model using recent text-to-video backbones. Experimental results demonstrate that RefVFX produces visually consistent and temporally coherent edits, generalizes across unseen effect categories, and outperforms prompt-only baselines in both quantitative metrics and human preference. See our website $\href{https://tuningfreevisualeffects-maker.github.io/Tuning-free-Visual-Effect-Transfer-across-Videos-Project-Page/}{at\ this\ URL}$.

Figures (9)

• Figure 1: Overview. We present RefVFX, a tuning-free framework for visual effect transfer across videos. Given a reference effect video and an input video, our method produces a new output video where the reference’s temporal effect is seamlessly applied to the input’s content and motion. Unlike prompt-based or keyframe-conditioned approaches, RefVFX directly learns to interpret and reproduce complex time-varying visual effects such as material transformations, object additions, or atmospheric effects from example videos during training, enabling faithful and coherent visual effect transfer at inference time.
• Figure 2: Dataset Examples. We show example triplets from each of our datasets. (a) We curate a reference video + image to video dataset by collecting Low Rank Adapters (LoRAs) hu2022lora for different Image to Video effects online. For each effect, we can apply its corresponding LoRA to two separate images to create a triplet. (b) We create a custom pipeline for generating text guided reference video + input video to video effects. For more details, see Figure \ref{['fig:v2v_method']}. (c) We generate a large scale set of (ref video, input video, output video) triplets by curating specific code pipelines that apply specific, detailed effects to arbitrary videos. Armed with a specific code based effect and a fixed set of hyperparameters, we can apply the exact effect to an arbitrary number of input videos to create many triplets.
• Figure 3: Dataset Subset. We display a summary of our dataset structure and available sample effects. These include LoRA-based Image-to-Video, our scalable V2V pipeline (see Fig. 4) , and programmatic methods. For code-based effects, individual tasks are created by combining specific effects (e.g., Glow) and transitions (e.g., Fade In) with randomized hyperparameters.
• Figure 4: Text-Guided Video Editing Pair Creation. We present a method to generate a pair of videos $(V, V')$ from an effect prompt $E$, where $V$ is an initial video and $V'$ is the video with an effect $E$ applied. First, an image generation model is used to create an initial image. Next, an image editing model is used to change the pose, camera angle, and facial expression of the image. Finally, this image is again edited to add effect $E$. The first and second generated images are used with a first last frame model to output video $V$. Then, we use a conditional video model conditioned on the original first frame, effect edited last frame, and intermediate poses from video $V$ as conditioning to create video $V'$.
• Figure 5: Architecture overview. (a) Standard Wan Video First–Last Frame to Video (FLF2V) architecture: noisy spatio-temporal latents are channel-wise concatenated with conditioning inputs and a mask, then patchified, embedded, and processed by the diffusion transformer to predict velocity. (b) In our setup, x input video latents are used as conditioning for the noisy latents, while reference video latents are concatenated width-wise to both. The latent mask is set to 1 for frames preserved exactly in the output and 0.5 for those to be modified; the reference latent mask is all ones. This design doubles the token count relative to base generation while conditioning on both the reference effect video and input video. Since all three inputs are channel-concatenated before patchification, repeated clean reference latents are merged channel-wise before embedding, ensuring no redundant reference information across tokens.