DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing

TL;DR

This work addresses the challenge of long-range temporal consistency in diffusion-based video editing for human-centric content with large motions and view changes. It introduces DynVideo-E, a video-NeRF framework that represents videos with a 3D dynamic human space and a 3D background, edited via a pose-guided deformation field and propagated through 3D deformation. The method combines reconstruction losses, 3D SDS from a 3D diffusion prior, 2D personalized SDS, text-guided local parts super-resolution, and background style transfer to achieve consistent, high-fidelity edits across views, demonstrated on HOSNeRF and NeuMan with strong improvements over state-of-the-art baselines. Ablation studies verify the importance of each component, and results show substantial gains in textual faithfulness and human preference, albeit with notable training-time requirements. Future work includes accelerating training via voxel/hash-grid representations while preserving quality.

Abstract

Despite recent progress in diffusion-based video editing, existing methods are limited to short-length videos due to the contradiction between long-range consistency and frame-wise editing. Prior attempts to address this challenge by introducing video-2D representations encounter significant difficulties with large-scale motion- and view-change videos, especially in human-centric scenarios. To overcome this, we propose to introduce the dynamic Neural Radiance Fields (NeRF) as the innovative video representation, where the editing can be performed in the 3D spaces and propagated to the entire video via the deformation field. To provide consistent and controllable editing, we propose the image-based video-NeRF editing pipeline with a set of innovative designs, including multi-view multi-pose Score Distillation Sampling (SDS) from both the 2D personalized diffusion prior and 3D diffusion prior, reconstruction losses, text-guided local parts super-resolution, and style transfer. Extensive experiments demonstrate that our method, dubbed as DynVideo-E, significantly outperforms SOTA approaches on two challenging datasets by a large margin of 50% ~ 95% for human preference. Code will be released at https://showlab.github.io/DynVideo-E/.

Figures (10)

• Figure 1: Given a reference subject image and a background style image, our DynVideo-E enables highly consistent editing of large-scale motion- and view-change human-centric videos (a-c).
• Figure 2: Overview of DynVideo-E. (1) Our video-NeRF model represents the input video as a 3D dynamic human space coupled with the deformation field and a 3D static background space. (2) Orange flowchart: Given the reference subject image, we edit the animatable 3D dynamic human space under multi-view multi-pose configurations by leveraging reconstruction losses, 2D personalized diffusion priors, 3D diffusion priors, and local parts super-resolution. (3) Green flowchart: A style transfer loss in feature spaces is utilized to transfer the reference style to our 3D background model. (4) Edited videos can be accordingly rendered by volume rendering in the edited video-NeRF model under source video camera poses, and we can also achieve high-fidelity free-viewpoint renderings of edited dynamic scenes.
• Figure 3: Qualitative comparisons of DynVideo-E against SOTA approaches on the Backpack scene (a) and Jogging scene (b).
• Figure 4: Qualitative ablation results of our method on each proposed component for (a) Backpack scene and (b) Lab scene.
• Figure 5: DynVideo-E network designs: (a) Editing Background model, (b) Original human-object model, (c) Editing human model.