Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language-Guided Face Animation by Recurrent StyleGAN-based Generator

Tiankai Hang, Huan Yang, Bei Liu, Jianlong Fu, Xin Geng, Baining Guo

TL;DR

...

Abstract

Recent works on language-guided image manipulation have shown great power of language in providing rich semantics, especially for face images. However, the other natural information, motions, in language is less explored. In this paper, we leverage the motion information and study a novel task, language-guided face animation, that aims to animate a static face image with the help of languages. To better utilize both semantics and motions from languages, we propose a simple yet effective framework. Specifically, we propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames. To optimize the proposed framework, three carefully designed loss functions are proposed including a regularization loss to keep the face identity, a path length regularization loss to ensure motion smoothness, and a contrastive loss to enable video synthesis with various language guidance in one single model. Extensive experiments with both qualitative and quantitative evaluations on diverse domains (\textit{e.g.,} human face, anime face, and dog face) demonstrate the superiority of our model in generating high-quality and realistic videos from one still image with the guidance of language. Code will be available at https://github.com/TiankaiHang/language-guided-animation.git.

Language-Guided Face Animation by Recurrent StyleGAN-based Generator

TL;DR

...

Abstract

Recent works on language-guided image manipulation have shown great power of language in providing rich semantics, especially for face images. However, the other natural information, motions, in language is less explored. In this paper, we leverage the motion information and study a novel task, language-guided face animation, that aims to animate a static face image with the help of languages. To better utilize both semantics and motions from languages, we propose a simple yet effective framework. Specifically, we propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames. To optimize the proposed framework, three carefully designed loss functions are proposed including a regularization loss to keep the face identity, a path length regularization loss to ensure motion smoothness, and a contrastive loss to enable video synthesis with various language guidance in one single model. Extensive experiments with both qualitative and quantitative evaluations on diverse domains (\textit{e.g.,} human face, anime face, and dog face) demonstrate the superiority of our model in generating high-quality and realistic videos from one still image with the guidance of language. Code will be available at https://github.com/TiankaiHang/language-guided-animation.git.
Paper Structure (28 sections, 10 equations, 12 figures, 1 table)

This paper contains 28 sections, 10 equations, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Formulation
  5. Cross-modal Encoder
  6. Recurrent Motion Generator
  7. Pre-trained Frame Synthesizer
  8. Loss Design
  9. Experiments
  10. Implementation Details
  11. Network Architecture
  12. Training Setting
  13. Datasets
  14. Baselines
  15. Metrics
  16. ...and 13 more sections

Figures (12)

  • Figure 1: Frames generated from a face image indicating "smiling" by three methods: StyleCLIP interpolation, MoCoGAN-HD, and ours. Optical flow teed2020raft visualizes the motion between adjacent frame. StyleCLIP interpolation keeps the consistency well while the motion is limited. Results from MoCoGAN-HD show large motion with unexpected jitter. Our method not only keeps the consistency but also presents the motion.
  • Figure 2: Overview of our proposed framework. We take a pair of image and text as input and embed them by respective encoders. A recurrent motion generator is used to extract motion codes. The visual embedding and motion are processed by visual and text-aware motion mapper. Then the frames are generated by a pre-trained frame synthesizer, i.e., StyleGAN. The synthesizer is fixed during training process
  • Figure 3: Example results of animation using our recurrent motion generator. Our results are qualitatively better than StyleCLIP interpolation. The first and third rows are from interpolation, the second and fourth rows are from our model.
  • Figure 4: Example results for comparison with MoCoGAN-HD. We sample latent codes from StyleGAN's pre-trained space and synthesize frames using our framework and MoCoGAN-HD. Our results show more vivid motion and better temporal consistency than the baseline.
  • Figure 5: Example results for loss ablation study. The guidance is "closing eyes". (a)-(d) demonstrate the superiority of our contrastive loss. Comparisons between (c) and (e),(f) illustrate better disentanglement of the path regularization loss. (Better zoom in to see the details)
  • ...and 7 more figures