Audio2Rig: Artist-oriented deep learning tool for facial animation

TL;DR

Audio2Rig addresses the challenge of producing studio-style facial and lip-sync animation by learning from production-rig data. It uses a three-network pipeline consisting of a Conditional Variational AutoEncoder (CVAE) to map rig controllers into a latent space, an AudioNet CNN-GRU to infer the latent code from a MelSpectrogram, and a KeyNet to schedule keys, all conditioned on an emotion vector of dimension $N=6$. The system operates in Maya and outputs keys on rig controllers, enabling retakes and easy artistic adjustments, while preserving the show's style and ensuring data privacy by training on studio data. Results show accurate mouth and tongue animation and visible emotion retention in the upper face, with ongoing work to improve eye/eyebrow motion and broaden emotion support.

Abstract

Creating realistic or stylized facial and lip sync animation is a tedious task. It requires lot of time and skills to sync the lips with audio and convey the right emotion to the character's face. To allow animators to spend more time on the artistic and creative part of the animation, we present Audio2Rig: a new deep learning based tool leveraging previously animated sequences of a show, to generate facial and lip sync rig animation from an audio file. Based in Maya, it learns from any production rig without any adjustment and generates high quality and stylized animations which mimic the style of the show. Audio2Rig fits in the animator workflow: since it generates keys on the rig controllers, the animation can be easily retaken. The method is based on 3 neural network modules which can learn an arbitrary number of controllers. Hence, different configurations can be created for specific parts of the face (such as the tongue, lips or eyes). With Audio2Rig, animators can also pick different emotions and adjust their intensities to experiment or customize the output, and have high level controls on the keyframes setting. Our method shows excellent results, generating fine animation details while respecting the show style. Finally, as the training relies on the studio data and is done internally, it ensures data privacy and prevents from copyright infringement.

Figures (2)

• Figure 1: Audio2Rig inference workflow
• Figure 2: Example of applied emotions. From left to right: happy, angry, surprised and sad.