Over++: Generative Video Compositing for Layer Interaction Effects

TL;DR

Over++ addresses the challenge of adding physically grounded environmental interactions to video composites without degrading the original content. It defines augmented compositing and presents a diffusion-based video inpainting framework capable of mask-guided and text-conditioned effect generation, supported by a tailored dataset with paired real/synthetic and unpaired data. The approach preserves scene content while producing diverse, controllable effects like shadows, dust, and splashes, and it is shown to outperform existing baselines in both qualitative and quantitative evaluations, including user studies. This work enables practical, editable VFX workflows for professional video compositing with limited training data and flexible guidance modalities.

Abstract

In professional video compositing workflows, artists must manually create environmental interactions-such as shadows, reflections, dust, and splashes-between foreground subjects and background layers. Existing video generative models struggle to preserve the input video while adding such effects, and current video inpainting methods either require costly per-frame masks or yield implausible results. We introduce augmented compositing, a new task that synthesizes realistic, semi-transparent environmental effects conditioned on text prompts and input video layers, while preserving the original scene. To address this task, we present Over++, a video effect generation framework that makes no assumptions about camera pose, scene stationarity, or depth supervision. We construct a paired effect dataset tailored for this task and introduce an unpaired augmentation strategy that preserves text-driven editability. Our method also supports optional mask control and keyframe guidance without requiring dense annotations. Despite training on limited data, Over++ produces diverse and realistic environmental effects and outperforms existing baselines in both effect generation and scene preservation.

Figures (14)

• Figure 1: Quantitative comparison. We evaluate effect generation performance on 24 videos at both the image and video levels. * indicates methods that require an edited first frame for reference, where we use the first frame of the ground-truth video $\mathcal{I}_{\text{gt}}$ with the added effects. Methods marked in gray require masks. Best results are highlighted in red, and second-best in orange. Please see SM for video results.
• Figure 2: Limitations of inpainting models for effects. Simply compositing foreground "over" background produces an input without effects. Inpainting models such as VACE vace require per-frame mask and may still fail to generate the desired effect. Our method successfully produces the target wake (far right).
• Figure 3: Over++ framework. Given an input composite video lacking environmental effects such as shadows or wakes ($\mathcal{I}_{\text{over}}$), and an optional binary mask indicating the target effect regions ($\mathcal{M}_{\text{effect}}$), our model Over++ generates desired effects within the specified regions ($\hat{\mathcal{I}}$). Training includes unpaired data by zeroing out the latent codes of $\mathcal{I}_{\text{over}}$ and $\mathcal{M}_{\text{effect}}$. (Text prompts $\mathcal{T}$ are not shown here for simplicity.)
• Figure 3: Ablation study (table). We evaluate the contribution of each data source by removing it from the training set and measuring the drop in performance across three CLIP-based metrics.
• Figure 4: Mask generation for training data. Given a training video with effects $\mathcal{I}_{\text{gt}}$, we construct a version without effects $\mathcal{I}_{\text{over}}$. The effect mask $\mathcal{M}_{\text{effect}}$ is derived by applying mask pruning to the difference image $\delta(\mathcal{I}_{\text{gt}}, \mathcal{I}_{\text{over}})$ to remove noise and artifacts. Top: Synthetic data with clean $\delta (\mathcal{I}_{\text{gt}}, \mathcal{I}_{\text{over}})$. Bottom: Real-world data has noisier $\delta (\mathcal{I}_{\text{gt}}, \mathcal{I}_{\text{over}})$, requiring additional cleanup.