Revising Second Order Terms in Deep Animation Video Coding

TL;DR

This work addresses the limitations of First Order Motion Model in handling head rotations for low-bitrate facial animation. It replaces per-keypoint Jacobians with a global rotation parameter $\phi$ and a scaling factor $scf$ (with optional per-KP shear) learned from head-pose and KP data, reducing P-frame bitrate while preserving motion fidelity. Additionally, it stabilizes adversarial training using Gradient Normalization and a self-supervised landmark loss, enabling larger adversarial losses and improved visual quality. Empirical results on VoxCeleb2 show competitive LPIPS, DISTS, and FID scores with reduced bitrate compared to OSFV, while maintaining low computational requirements, suggesting broader applicability to learning-based video codecs.

Abstract

First Order Motion Model is a generative model that animates human heads based on very little motion information derived from keypoints. It is a promising solution for video communication because first it operates at very low bitrate and second its computational complexity is moderate compared to other learning based video codecs. However, it has strong limitations by design. Since it generates facial animations by warping source-images, it fails to recreate videos with strong head movements. This works concentrates on one specific kind of head movements, namely head rotations. We show that replacing the Jacobian transformations in FOMM by a global rotation helps the system to perform better on items with head-rotations while saving 40% to 80% of bitrate on P-frames. Moreover, we apply state-of-the-art normalization techniques to the discriminator to stabilize the adversarial training which is essential for generating visually appealing videos. We evaluate the performance by the learned metics LPIPS and DISTS to show the success our optimizations.

Figures (6)

• Figure 1: Block diagram of encoder and decoder in red boxes. Input are I-frame and P-frame on the left, output is the generated P-frame on the right. DNNs are depicted with green outlines.
• Figure 2: Block diagram of the proposed discriminator optimization. Red blocks are needed to calculate the discriminator loss.
• Figure 3: LPIPS on test set. Lower means better performance.
• Figure 4: DISTS on test set. Lower means better performance.
• Figure 5: Fréchet inception distance (FID) on test set. Lower means better performance.