TransPixeler: Advancing Text-to-Video Generation with Transparency

TL;DR

This work tackles RGBA video generation, a challenging task due to scarce RGBA video data and the need for consistent alpha channels. It introduces TransPixeler, a method that extends pretrained diffusion-transformer video models to jointly generate RGB and alpha channels by adding alpha tokens, using a shared positional encoding, a learnable domain embedding, and LoRA fine-tuning applied only to alpha tokens. The approach analyzes attention interactions to retain RGB quality while enabling RGB↔Alpha refinement, demonstrating strong RGB–alpha alignment with limited data. Experimental results show diverse, coherent RGBA videos and favorable alignment compared to generation-then-prediction baselines, suggesting significant potential for VFX, AR/VR, and interactive content creation.

Abstract

Text-to-video generative models have made significant strides, enabling diverse applications in entertainment, advertising, and education. However, generating RGBA video, which includes alpha channels for transparency, remains a challenge due to limited datasets and the difficulty of adapting existing models. Alpha channels are crucial for visual effects (VFX), allowing transparent elements like smoke and reflections to blend seamlessly into scenes. We introduce TransPixeler, a method to extend pretrained video models for RGBA generation while retaining the original RGB capabilities. TransPixar leverages a diffusion transformer (DiT) architecture, incorporating alpha-specific tokens and using LoRA-based fine-tuning to jointly generate RGB and alpha channels with high consistency. By optimizing attention mechanisms, TransPixar preserves the strengths of the original RGB model and achieves strong alignment between RGB and alpha channels despite limited training data. Our approach effectively generates diverse and consistent RGBA videos, advancing the possibilities for VFX and interactive content creation.

Figures (11)

• Figure 1: RGBA Video Generation with TransPixeler. By introducing LoRA layers into DiT-based text-to-video model with a novel alpha channel adaptive attention mechanism, our method enables RGBA video generation from text while preserving Text-to-Video quality.
• Figure 2: Comparison between Generation-Then-Prediction and our Joint Generation approach. Given the generated RGB in (a), (b) and (c) show the predicted alpha (top) and the composited result (bottom). In (d), the top shows the jointly generated alpha.
• Figure 3: Pipeline of TransPixeler. Our method is organized as follows: (1) Left: we extend the input of DiT to include new alpha tokens; (2) Top Center: we initialize alpha tokens with our positional encoding; (3) Bottom Center: we insert a partial LoRA and adjust attention computation during training and inference.
• Figure 4: Positional Encoding Design for RGBA Generation. Assigning alpha tokens the same positional encoding as RGB yields similar results, resulting in faster convergence after 1000 iterations compared to standard encoding strategies.
• Figure 5: Attention Rectification. (a) Eliminating all attention from alpha as a key preserves 100% RGB generation but leads to poor alignment. (b) Retaining all attention significantly degrades quality, causing a lack of motion in bicycles. (c) Our method achieves an effective balance.