EasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decomposition

TL;DR

EasyOmnimatte addresses slow, multi-stage video omnimatte pipelines by reusing a pretrained video inpainting diffusion model to predict the foreground alpha matte and its effects in one feed-forward pass. It introduces a Dual-Expert architecture with an Effect Expert (LoRA on late DiT blocks) and a Quality Expert (LoRA on all blocks), and a dual-timestep sampling strategy that uses the Effect Expert at high-noise steps and the Quality Expert at low-noise steps, reducing compute while preserving effects. A block-wise analysis reveals which DiT blocks encode effect cues, enabling targeted adaptation, and an end-to-end formulation directly computes $(F, \alpha,B)$ from $(V,M,c)$ with $V = \alpha F + (1-\alpha) B$. The approach achieves state-of-the-art results in quality and efficiency for video omnimatte and enables practical editing tasks such as shadows and reflections separation.

Abstract

Existing video omnimatte methods typically rely on slow, multi-stage, or inference-time optimization pipelines that fail to fully exploit powerful generative priors, producing suboptimal decompositions. Our key insight is that, if a video inpainting model can be finetuned to remove the foreground-associated effects, then it must be inherently capable of perceiving these effects, and hence can also be finetuned for the complementary task: foreground layer decomposition with associated effects. However, although naïvely finetuning the inpainting model with LoRA applied to all blocks can produce high-quality alpha mattes, it fails to capture associated effects. Our systematic analysis reveals this arises because effect-related cues are primarily encoded in specific DiT blocks and become suppressed when LoRA is applied across all blocks. To address this, we introduce EasyOmnimatte, the first unified, end-to-end video omnimatte method. Concretely, we finetune a pretrained video inpainting diffusion model to learn dual complementary experts while keeping its original weights intact: an Effect Expert, where LoRA is applied only to effect-sensitive DiT blocks to capture the coarse structure of the foreground and associated effects, and a fully LoRA-finetuned Quality Expert learns to refine the alpha matte. During sampling, Effect Expert is used for denoising at early, high-noise steps, while Quality Expert takes over at later, low-noise steps. This design eliminates the need for two full diffusion passes, significantly reducing computational cost without compromising output quality. Ablation studies validate the effectiveness of this Dual-Expert strategy. Experiments demonstrate that EasyOmnimatte sets a new state-of-the-art for video omnimatte and enables various downstream tasks, significantly outperforming baselines in both quality and efficiency.

Figures (13)

• Figure 1: EasyOmnimatte is the first, end-to-end video omnimatte method, eliminating the need for slow, multi-stage optimization-based pipelines (e.g., Gen-Omnimatte lee2025generative). EasyOmnimatte produces high-fidelity alpha mattes that accurately capture the associated effects, all within just seconds, representing a significant gain over existing state-of-the-art methods in both quality and efficiency.
• Figure 2: EasyOmnimatte. a) We branch out LoRA-finetuned blocks from the original inpainting DiT blocks to jointly predict the alpha matte, alongside the pretrained model. In each Branch DiT Block, LoRA are applied only to the duplicated set of input tokens, leaving the original inpainting branch unaffected. b) During sampling, the Effect Expert model is only employed at early, high-noise stages to generate coarse, effect-aware omnimatte predictions, while the Quality Expert model refines the alpha matte only at later, low-noise stages. This alternating strategy achieves high-quality results with greatly reduced compute cost, compared to individually sampling.
• Figure 3: Our block-wise Analysis.Bottom: Contribution scores per layer reveal the three inpainting stages. Top: Averaged attention maps ($\bm{S}_b$) for each stage show that the middle stage is most sensitive to effects (e.g., shadows), which are then suppressed in the final stage, corroborating our perception-then-elimination hypothesis.
• Figure 4: Qualitative comparison with optimization-based omnimatte method lee2025generative. Please zoom in for a better view.
• Figure 5: Qualitative comparisons with learning-based matting method lin2021realyang2025matanyone. Please zoom in for a better view.