Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VASE: Object-Centric Appearance and Shape Manipulation of Real Videos

Elia Peruzzo, Vidit Goel, Dejia Xu, Xingqian Xu, Yifan Jiang, Zhangyang Wang, Humphrey Shi, Nicu Sebe

TL;DR

VASE tackles object-centric video editing by decoupling appearance and shape control for a single foreground object. It extends a pre-trained image-conditioned diffusion model with temporal layers and shaping guidance via ControlNet, augmented by Joint Flow-Structure Augmentation, a Warping Flow-Completion Net, and a Segmentation Head to enforce explicit shape edits. The approach achieves competitive image-driven editing quality and unlocks novel shape-editing capabilities without per-video optimization, demonstrated on real videos. Limitations include occlusions and challenging perspective changes, pointing toward future 3D-aware enhancements for robustness.

Abstract

Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models. However, most of these methods holistically edit the frame using the text, exploiting the prior given by foundation diffusion models and focusing on improving the temporal consistency across frames. In this work, we introduce a framework that is object-centric and is designed to control both the object's appearance and, notably, to execute precise and explicit structural modifications on the object. We build our framework on a pre-trained image-conditioned diffusion model, integrate layers to handle the temporal dimension, and propose training strategies and architectural modifications to enable shape control. We evaluate our method on the image-driven video editing task showing similar performance to the state-of-the-art, and showcasing novel shape-editing capabilities. Further details, code and examples are available on our project page: https://helia95.github.io/vase-website/

VASE: Object-Centric Appearance and Shape Manipulation of Real Videos

TL;DR

VASE tackles object-centric video editing by decoupling appearance and shape control for a single foreground object. It extends a pre-trained image-conditioned diffusion model with temporal layers and shaping guidance via ControlNet, augmented by Joint Flow-Structure Augmentation, a Warping Flow-Completion Net, and a Segmentation Head to enforce explicit shape edits. The approach achieves competitive image-driven editing quality and unlocks novel shape-editing capabilities without per-video optimization, demonstrated on real videos. Limitations include occlusions and challenging perspective changes, pointing toward future 3D-aware enhancements for robustness.

Abstract

Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models. However, most of these methods holistically edit the frame using the text, exploiting the prior given by foundation diffusion models and focusing on improving the temporal consistency across frames. In this work, we introduce a framework that is object-centric and is designed to control both the object's appearance and, notably, to execute precise and explicit structural modifications on the object. We build our framework on a pre-trained image-conditioned diffusion model, integrate layers to handle the temporal dimension, and propose training strategies and architectural modifications to enable shape control. We evaluate our method on the image-driven video editing task showing similar performance to the state-of-the-art, and showcasing novel shape-editing capabilities. Further details, code and examples are available on our project page: https://helia95.github.io/vase-website/
Paper Structure (16 sections, 5 equations, 10 figures, 3 tables)

This paper contains 16 sections, 5 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Backbone
  5. Joint Flow-Structure Augmentation (JFSA)
  6. Warping Flow-Completion Net (wFCN)
  7. Segmentation Head (SH)
  8. Implementation Details
  9. Experiments
  10. Image-Driven Appearance Editing
  11. Joint Appearance-Shape Editing
  12. Ablation
  13. Conclusions
  14. VASE components
  15. Model and Implementation Details
  16. ...and 1 more sections

Figures (10)

  • Figure 1: With VASE, we can manipulate both the shape and appearance of an object within a real video. In the top row, we present the source video, followed by a series of edits showcasing precise adjustments in both attributes. In the first frame (1st column), the keyframe structure is overlayed to the object. Red areas represent regions that have been removed from the original shape, while green regions denote newly added parts. The driver image for appearance edits is showcased in the bottom left corner.
  • Figure 2: We enhance video editing by conditioning the synthesis of the video on two branches, one that controls the appearance and the other responsible for the motion and the structure of the object. To enable shape modifications, we propose a Joint Flow-Structure Augmentation pipeline that outputs an augmented flow which is processed by a Flow-Completion Network, before going as input to the final ControlNet module. Furthermore, we introduce an auxiliary loss, used to enhance the model fidelity to the input segmentation map.
  • Figure 3: Results for the Image Driven Appearance Editing task. The driver image is displayed in the bottom-left corner of the initial frame in the source video. We refer the reader to the Supp. Mat. for video results that better showcase the differences.
  • Figure 4: Results for the Joint Appearance-Shape Editing task. Please, note that the shape edit is provided only for the first keyframe, we show all of them only for visualization purposes. We highlight the structure modifications either in red or green.
  • Figure 5: Qualitative results obtained with different versions of our model.
  • ...and 5 more figures