VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs

TL;DR

VidStyleODE tackles the challenge of disentangling content, motion, and style in videos by encoding content in the StyleGAN2 ${\mathcal{W}}_+$ space and modeling motion with a latent NeuralODE. Frames are generated as latent residuals ${\Delta\mathbf{z}}_t$ added to a global content code ${\mathbf{z}}_C$, with dynamics ${\mathbf{z}}_d$ evolved via a latent ODE and conditioning through attention on an external style cue ${\mathbf{z}}_S$ derived from CLIP-guided text. The model is trained with a non-adversarial loss that combines a CLIP-based temporal consistency term, appearance and structure reconstruction losses, a CLIP directional loss, and a latent-direction regularizer, enabling text-driven edits and motion transfer on high-resolution outputs. Experiments on Fashion Videos and RAVDESS show strong temporal coherence, competitive perceptual quality, and the ability to interpolate/extrapolate and localize motion dynamics, all while keeping the source identity intact. The work demonstrates that a frozen StyleGAN generator combined with latent ODE dynamics can achieve flexible, high-quality video editing without adversarial training, with potential for broader text-guided video manipulation tasks.

Abstract

We propose $\textbf{VidStyleODE}$, a spatiotemporally continuous disentangled $\textbf{Vid}$eo representation based upon $\textbf{Style}$GAN and Neural-$\textbf{ODE}$s. Effective traversal of the latent space learned by Generative Adversarial Networks (GANs) has been the basis for recent breakthroughs in image editing. However, the applicability of such advancements to the video domain has been hindered by the difficulty of representing and controlling videos in the latent space of GANs. In particular, videos are composed of content (i.e., appearance) and complex motion components that require a special mechanism to disentangle and control. To achieve this, VidStyleODE encodes the video content in a pre-trained StyleGAN $\mathcal{W}_+$ space and benefits from a latent ODE component to summarize the spatiotemporal dynamics of the input video. Our novel continuous video generation process then combines the two to generate high-quality and temporally consistent videos with varying frame rates. We show that our proposed method enables a variety of applications on real videos: text-guided appearance manipulation, motion manipulation, image animation, and video interpolation and extrapolation. Project website: https://cyberiada.github.io/VidStyleODE

Figures (24)

• Figure 1: VidStyleODE provides a spatiotemporal video representation in which motion and content info are disentangled, making it ideal for: (a) animating images, (b) consistent video appearance manipulation based on text, (c) body part motion transfer ([blue] boxes) from a co-driving video while preserving remaining driving video dynamics ([orange] boxes) intact, (d) temporal interpolation, and (e) extrapolation. Zoom in for better viewing.
• Figure 2: VidStyleODE overview. We encode video dynamics and process them using a ConvGRU layer to obtain a dynamic latent representation $\mathbf{Z}_{d0}$ used to initialize a latent ODE of the motion (bottom). We also encode the video in $\mathcal{W}_+$ space to obtain a global latent code $Z_C$ (middle). We combine the two with an external style cue through an attention mechanism to condition the AdaIN layer that predicts the directions to the latent codes of the frames in the target video (top). Modules in gray are pre-trained and frozen during training.
• Figure 3: Proposed attention scheme utilized in VidStyleODE.
• Figure 4: Text-guided editing results. VidStyleODE lets the users manipulate a frame based on a text prompt, and transfer manipulated attributes to other videos in a consistent way. Source frames are shown at the top left corner along with the target texts.
• Figure 5: Qualitative comparison against the state-of-the-art. VidStyleODE produces more realistic results than existing semantic video methods when changing sleeve length from short to long, with improved visual quality and manipulation accuracy. HairCLIP, a frame-level method, lacks temporal coherence.