TransVDM: Motion-Constrained Video Diffusion Model for Transparent Video Synthesis

TL;DR

This work tackles transparent video generation with diffusion models by introducing TransVDM, which encodes alpha information into the latent space via a Transparent Variational Autoencoder (TVAE) and mitigates artifacts in transparent regions with an Alpha Motion Constraint Module (AMCM). The framework combines a TVAE with a pretrained UNet-based VDM and trains on a newly curated dataset of 250k transparent frames, using a two-stage process (TVAE then AMCM). Experimental results, including qualitative comparisons and ablation studies, demonstrate improved temporal fidelity and reduced artifacts over RGB-only baselines and post-processing approaches. The work suggests practical impact for applications requiring end-to-end transparent video synthesis and outlines paths for more controllable generation and stronger diffusion backbones.

Abstract

Recent developments in Video Diffusion Models (VDMs) have demonstrated remarkable capability to generate high-quality video content. Nonetheless, the potential of VDMs for creating transparent videos remains largely uncharted. In this paper, we introduce TransVDM, the first diffusion-based model specifically designed for transparent video generation. TransVDM integrates a Transparent Variational Autoencoder (TVAE) and a pretrained UNet-based VDM, along with a novel Alpha Motion Constraint Module (AMCM). The TVAE captures the alpha channel transparency of video frames and encodes it into the latent space of the VDMs, facilitating a seamless transition to transparent video diffusion models. To improve the detection of transparent areas, the AMCM integrates motion constraints from the foreground within the VDM, helping to reduce undesirable artifacts. Moreover, we curate a dataset containing 250K transparent frames for training. Experimental results demonstrate the effectiveness of our approach across various benchmarks.

Figures (3)

• Figure 1: TransVDM Framework. The TVAE is trained during Stage 1, which is utilized in Stage 2 to produce adjusted noisy latents with alpha information. In Stage 1, the RGB channels receive a smoothing operation to mitigate abrupt visual artifacts that may occur when alpha edges are inaccurately predicted, allowing the transparent areas of the original image to blend more seamlessly. The Stage 2 diagram illustrates a U-Net based VDM, the concatenation of the conditioned frame and adjusted noisy latents, along with a CLIP text encoder and a simplified processing flow of the AMCM module. The AMCM module actually comprises fully connected layers and a temporal attention layer. The input features from the temporal transformer are combined with $B$, where B represents the normalized coordinates of the bounding boxes for each input frame, and processed through the AMCM module before being passed back to the temporal transformer.
• Figure 2: Qualitative Results. The input includes two image-text pairs, with the corresponding texts being "a man smiling" and "the candle is lit up".
• Figure 3: More cases generated by TransVDM, with the first column showing the conditioned images and the following columns displaying the generated outputs. The prompts are "a cat turns head", "Barbie blinking her eyes", and "the apple is swaying in the wind".