InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation

TL;DR

InMoDeGAN introduces a novel unconditional video generator that explicitly decomposes motion into semantic, interpretable components via a motion bank with an orthogonal basis. The architecture combines a two-stream generator (appearance and motion paths) with a Temporal Pyramid Discriminator and a Linear Motion Decomposition framework to achieve high-quality videos while enabling controllable motion via identified motion directions. A dedicated interpretability evaluation using optical flow demonstrates that certain directions correspond to semantically meaningful motions (e.g., mouth vs head movement, robot arm directions), enabling targeted manipulation. Across VoxCeleb2-mini and BAIR-robot, InMoDeGAN achieves state-of-the-art video quality and showcases the ability to generate longer and higher-resolution videos, suggesting practical utility for video editing, synthesis, and data augmentation.

Abstract

In this work, we introduce an unconditional video generative model, InMoDeGAN, targeted to (a) generate high quality videos, as well as to (b) allow for interpretation of the latent space. For the latter, we place emphasis on interpreting and manipulating motion. Towards this, we decompose motion into semantic sub-spaces, which allow for control of generated samples. We design the architecture of InMoDeGAN-generator in accordance to proposed Linear Motion Decomposition, which carries the assumption that motion can be represented by a dictionary, with related vectors forming an orthogonal basis in the latent space. Each vector in the basis represents a semantic sub-space. In addition, a Temporal Pyramid Discriminator analyzes videos at different temporal resolutions. Extensive quantitative and qualitative analysis shows that our model systematically and significantly outperforms state-of-the-art methods on the VoxCeleb2-mini and BAIR-robot datasets w.r.t. video quality related to (a). Towards (b) we present experimental results, confirming that decomposed sub-spaces are interpretable and moreover, generated motion is controllable.

Figures (8)

• Figure 1: Controllable video generation. InMoDeGAN learns to decompose motion into semantic motion-components. This allows for manipulations in the latent code to invoke motion in generated videos that is human interpretable. Top (a) robot arm moves backwards, bottom (a) robot arm moves to the right. Similarly, in (b) we are animating the face to 'talk' (top) and 'move head' (bottom).
• Figure 2: InMoDeGAN-architecture. InMoDeGAN comprises of a Generator and a two-stream Discriminator. We design the architecture of the Generator based on proposed Linear Motion Decomposition. Specifically, a motion bank is incorporated in the Generator to learn and store a motion dictionary $D$, which contains motion-directions $[d_0,d_1,..,d_{N-1}]$. We use an appearance net $G_A$ to map appearance noise $z_a$ into a latent code $w_0$, which serves as the initial latent code of a generated video. A motion net $G_M$ maps a sequence of motion noises $\{z_{m_t}\}^{T-1}_{t=1}$ into a sequence $\{A_t\}^{T-1}_{t=1}$, which represent motion magnitudes. Each latent code $w_t$ is computed based on Linear Motion Decomposition using $w_0$, $D$ and $A_t$. Generated video $V$ is obtained by a synthesis net $G_S$ that maps the sequence of latent codes $\{w_t\}^{T-1}_{t=0}$ into an image sequence $\{x_t\}^{T-1}_{t=0}$. Our discriminator comprises an image discriminator $D_I$ and a Temporal Pyramid Discriminator (TPD) that contains several video discriminators $D_{V_i}$, leveraging different temporal speeds $\upsilon_i$ to improve generated video quality. While $D_I$ accepts as input a randomly sampled image per video, each $D_{V_i}$ is accountable for one temporal resolution.
• Figure 3: Analysis of $\alpha$. (a) Mean and variance bar charts, indicating top 10 motion-directions with highest values in $A_{\Bar{t}}$. (b) Time v.s. $\alpha$. Each figure represents a video sample. We illustrate two samples from BAIR-robot (top) and two from VoxCeleb2-mini (bottom). Top 5 dimensions in $\alpha$ are plotted in different color.
• Figure 4: Directions analysis on BAIR-robot. A generated video sample, related optical flow images (top), activation of only$d_1$ (middle), and activation of only$d_{511}$ (bottom). Optical flow images indicate that $d_1$ is accountable for moving the robot arm backward, whereas $d_{511}$ for moving it left and right.
• Figure 5: Optical flow quantization. (a) Middlebury colorwheel, (b) $\lambda(x_{t,j})$ and H on the colorwheel, (c) one frame from BAIR-robot and (d) related optical flow.