DeX-Portrait: Disentangled and Expressive Portrait Animation via Explicit and Latent Motion Representations

TL;DR

DeX-Portrait tackles the challenge of disentangled head pose and facial expression control in portrait animation by combining an explicit pose representation (RTS) with a latent expression code. A GAN-based Disentangled Motion Trainer learns separate pose and expression encoders, which are then injected into a diffusion model via a dual-branch pose mechanism and cross-attention for expression, capped with progressive CFG to preserve identity. The approach achieves expressive, high-fidelity animation with improved disentanglement over state-of-the-art baselines on multiple datasets and enables pose-only or expression-only editing. This has practical impact for controllable, high-quality portrait animation while emphasizing responsible use and ethical considerations.

Abstract

Portrait animation from a single source image and a driving video is a long-standing problem. Recent approaches tend to adopt diffusion-based image/video generation models for realistic and expressive animation. However, none of these diffusion models realizes high-fidelity disentangled control between the head pose and facial expression, hindering applications like expression-only or pose-only editing and animation. To address this, we propose DeX-Portrait, a novel approach capable of generating expressive portrait animation driven by disentangled pose and expression signals. Specifically, we represent the pose as an explicit global transformation and the expression as an implicit latent code. First, we design a powerful motion trainer to learn both pose and expression encoders for extracting precise and decomposed driving signals. Then we propose to inject the pose transformation into the diffusion model through a dual-branch conditioning mechanism, and the expression latent through cross attention. Finally, we design a progressive hybrid classifier-free guidance for more faithful identity consistency. Experiments show that our method outperforms state-of-the-art baselines on both animation quality and disentangled controllability.

Figures (13)

• Figure 1: Left: Given an arbitrary source portrait image, driving expression images, and driving pose images, DeX-Portrait achieves disentangled and expressive portrait animation. Middle: DeX-Portrait enables expression-only and pose-only editing while keeping the other driving signal exactly the same as the source. Right: Compared with the state-of-the-art work, X-NeMo xnemo, our method offers superior fine-grained control over head pose including rotation, translation and scale.
• Figure 2: Our pipeline consists of two stages: (a) Training a disentangled pose and expression encoder using a motion trainer. (b) Taming a latent diffusion model for disentangled and expressive portrait animation.
• Figure 3: Illustration of the pose and expression augmentation.
• Figure 4: Illustration of the ray map of head pose.
• Figure 5: Pose-only generation preserves strong identity consistency (second from right). Compared with the original CFG, our method achieves better consistency with the source portrait (e.g., facial shapes) in scenarios involving significant pose and expression variations.