Vectorized Video Representation with Easy Editing via Hierarchical Spatio-Temporally Consistent Proxy Embedding

TL;DR

This work tackles the instability of pixel-level priors in video representations by introducing hierarchical spatio-temporally consistent proxy nodes that encode motion and appearance across multiple semantic layers. Proxies are generated from spatially vectorized video regions and then propagated with dynamic augmentation to maintain temporal coherence, handling occlusions and large motions without heavy reliance on dense optical flow. Appearance is embedded implicitly on proxy nodes via per-frame triangulation and high-frequency decoding, enabling efficient reconstruction and controllable editing by moving node positions and features. Experiments on DAVIS demonstrate high reconstruction quality with fewer parameters and show effective video in-painting, editing, and spatio-temporal interpolation, highlighting robustness, efficiency, and practical impact for video editing workflows.

Abstract

Current video representations heavily rely on unstable and over-grained priors for motion and appearance modelling, \emph{i.e.}, pixel-level matching and tracking. A tracking error of just a few pixels would lead to the collapse of the visual object representation, not to mention occlusions and large motion frequently occurring in videos. To overcome the above mentioned vulnerability, this work proposes spatio-temporally consistent proxy nodes to represent dynamically changing objects/scenes in the video. On the one hand, the hierarchical proxy nodes have the ability to stably express the multi-scale structure of visual objects, so they are not affected by accumulated tracking error, long-term motion, occlusion, and viewpoint variation. On the other hand, the dynamic representation update mechanism of the proxy nodes adequately leverages spatio-temporal priors of the video to mitigate the impact of inaccurate trackers, thereby effectively handling drastic changes in scenes and objects. Additionally, the decoupled encoding manner of the shape and texture representations across different visual objects in the video facilitates controllable and fine-grained appearance editing capability. Extensive experiments demonstrate that the proposed representation achieves high video reconstruction accuracy with fewer parameters and supports complex video processing tasks, including video in-painting and keyframe-based temporally consistent video editing.

Figures (12)

• Figure 1: Illustration of our Motivation. Current video representations depend heavily on low-level, unstable priors for motion and appearance modeling, such as pixel-wise matching and tracking, which introduces representation errors and impairs performance of downstream tasks like video editing, especially on videos with occlusions and large motions. This work proposes encoding multi-scale local structures using hierarchical spatio-temporally consistent proxy nodes, and mitigating tracking errors by operating on the trajectories of sparse proxies, which achieves stable and accurate representation and supports complex video processing tasks.
• Figure 2: Overview of our framework. We decompose the input video into semantic layers, and then embed video motion and appearance into hierarchical spatio-temporally consistent proxy nodes through a dynamic proxy node supplementation and propagation mechanism. Our representation enables various video processing tasks. The dashed arrows indicate the propagation of corresponding color-coded supplementary nodes from current frame to target frame via the tracking algorithm.
• Figure 3: Qualitative comparisons on DAVIS. Our method achieves high-fidelity reconstruction with intricate texture details, especially on complex scenes. Please zoom in for more details. More visualizations are provided in the supplementary material.
• Figure 4: Qualitative results of video in-painting. Our method effectively completes the background even during complex scene transitions. Please zoom in for detailed comparisons. More visualizations are provided in the supplementary material.
• Figure 5: Qualitative results of image-based consistent editing. Our method ensures stable and controlled image editing propagation across the entire video, even in cases of large-scale motion, outperforming compared methods. Please zoom in for details.