Latent Image Animator: Learning to Animate Images via Latent Space Navigation

TL;DR

Latent Image Animator (LIA) tackles image animation without explicit structure priors by learning a latent-space motion path through Linear Motion Decomposition with an orthogonal basis. An autoencoder jointly learns appearance and motion representations, decoding latent paths into multi-scale flow fields to warp the source image toward driving motion in a self-supervised framework. Across VoxCeleb, TaichiHD, and TED-talk, LIA achieves higher fidelity and robustness than state-of-the-art methods, including cross-dataset transfers to unseen identities, with a transparent, interpretable motion dictionary. This approach reduces model complexity, enhances generalization, and points to a promising direction for interpretable latent-space videography.

Abstract

Due to the remarkable progress of deep generative models, animating images has become increasingly efficient, whereas associated results have become increasingly realistic. Current animation-approaches commonly exploit structure representation extracted from driving videos. Such structure representation is instrumental in transferring motion from driving videos to still images. However, such approaches fail in case the source image and driving video encompass large appearance variation. Moreover, the extraction of structure information requires additional modules that endow the animation-model with increased complexity. Deviating from such models, we here introduce the Latent Image Animator (LIA), a self-supervised autoencoder that evades need for structure representation. LIA is streamlined to animate images by linear navigation in the latent space. Specifically, motion in generated video is constructed by linear displacement of codes in the latent space. Towards this, we learn a set of orthogonal motion directions simultaneously, and use their linear combination, in order to represent any displacement in the latent space. Extensive quantitative and qualitative analysis suggests that our model systematically and significantly outperforms state-of-art methods on VoxCeleb, Taichi and TED-talk datasets w.r.t. generated quality.

Figures (11)

• Figure 1: LIA animation examples. The two images of Marilyn Monroe and Emmanuel Macron are animated by LIA, which transfers motion of a driving video (smaller images on the top) from VoxCeleb dataset Chung18b onto the still images. LIA is able to successfully animate these two images without relying on any explicit structure representations, such as landmarks and region representations.
• Figure 2: General pipeline. Our objective is to transfer motion via latent space navigation. The entire training pipeline consists of two steps. Firstly, we encode a source image $x_s$ into a latent code $z_{s\rightarrow r}$. By linearly navigating $z_{s\rightarrow r}$ along a path $w_{r\rightarrow d}$, we reach a target latent code $z_{s\rightarrow d}$. The latent paths are represented by a linear combination between a set of learned motion directions (e.g., $d_{1}$ and $d_{2}$), which is an orthogonal basis, and associated magnitudes. In the second step, we decode $z_{s\rightarrow d}$ to a target dense optical flow field $\phi_{s\rightarrow d}$, which is used to warp $x_s$ into the driving image $x_d$. While we train our model using images from the same video sequence, in the testing phase, $x_s$ and $x_d$ generally pertain to different identities.
• Figure 3: Overview of LIA. LIA is an autoencoder consisting of two networks, an encoder $E$ and a generator $G$. In the latent space, we apply Linear Motion Decomposition (LMD) towards learning a motion dictionary $D_m$, which is an orthogonal basis where each vector represents a basic visual transformation. LIA takes two frames sampled from the same video sequence as source image $x_s$ and driving image $x_d$ respectively during training. Firstly, it encodes $x_s$ into a source latent code $z_{s\rightarrow r}$ and $x_d$ into a magnitude vector $A_{r\rightarrow d}$. Then, it linearly combines $A_{r\rightarrow d}$ and a trainable $D_m$ using LMD to obtain a latent path $w_{r\rightarrow d}$, which is used to navigate $z_{s_\rightarrow r}$ to a target code $z_{s\rightarrow d}$. Finally, $G$ decodes $z_{s\rightarrow d}$ into a target dense flow field and warps $x_s$ to an output image $x_{s\rightarrow d}$. The training objective is to reconstruct $x_d$ using $x_{s\rightarrow d}$.
• Figure 4: Qualitative results. Examples for same-dataset absolute motion transfer on TaichiHD (top-right) and TED-talk (bottom-right). On VoxCeleb (left), we demonstrate cross-dataset relative motion transfer. We successfully transfer motion between $x_1$ and $x_t$ from videos in VoxCeleb to $x_s$ from FFHQ, the latter not being used for training.
• Figure 5: Visualization of reference images. Example source (top) and reference images (down) from VoxCeleb, TaichiHD and TED-talk datasets. Our network learns reference images of a consistently frontal pose, systematically for all input images of each dataset.