FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers

TL;DR

FantasyPortrait tackles expressive portrait animation for single and multi-character settings without relying on explicit geometry priors. It leverages a diffusion-transformer in latent space with expression-augmented implicit control to capture subtle lip and emotional dynamics. A masked cross-attention mechanism enables independent yet coordinated control across multiple characters. The paper introduces Multi-Expr dataset and ExprBench to support training and evaluation. Experiments show state-of-the-art performance in both quantitative metrics and human judgments, particularly in cross-identity reenactment and multi-character scenarios.

Abstract

Producing expressive facial animations from static images is a challenging task. Prior methods relying on explicit geometric priors (e.g., facial landmarks or 3DMM) often suffer from artifacts in cross reenactment and struggle to capture subtle emotions. Furthermore, existing approaches lack support for multi-character animation, as driving features from different individuals frequently interfere with one another, complicating the task. To address these challenges, we propose FantasyPortrait, a diffusion transformer based framework capable of generating high-fidelity and emotion-rich animations for both single- and multi-character scenarios. Our method introduces an expression-augmented learning strategy that utilizes implicit representations to capture identity-agnostic facial dynamics, enhancing the model's ability to render fine-grained emotions. For multi-character control, we design a masked cross-attention mechanism that ensures independent yet coordinated expression generation, effectively preventing feature interference. To advance research in this area, we propose the Multi-Expr dataset and ExprBench, which are specifically designed datasets and benchmarks for training and evaluating multi-character portrait animations. Extensive experiments demonstrate that FantasyPortrait significantly outperforms state-of-the-art methods in both quantitative metrics and qualitative evaluations, excelling particularly in challenging cross reenactment and multi-character contexts. Our project page is https://fantasy-amap.github.io/fantasy-portrait/.

Figures (5)

• Figure 1: Given a portrait image and a reference motion video, FantasyPortrait generates vivid animated portraits during cross-reenactment. It achieves high-fidelity facial dynamics and natural head movements for both single-character and multi-character.
• Figure 2: Overview of FantasyPortrait.
• Figure 3: Examples of ExprBench.
• Figure 4: Qualitative Results.
• Figure 5: Qualitative Ablation Results.