Instruction-Driven 3D Facial Expression Generation and Transition

TL;DR

This work tackles instruction-driven 3D facial expression generation and transition from a single RGB image. It introduces two core components: IFED, a multimodal cross-attention module that aligns text prompts with facial expression parameters, and I2FET, a conditional variational framework that maps text and latent factors to expression/pose trajectories; FLAME/DECA-based rendering completes the pipeline. Evaluations on CK+ and CelebV-HQ show superior transition accuracy and high-quality rendering compared to baselines, with extensive ablations validating the roles of IFED, CAFT, and specialized losses. The approach enables diverse, text-guided facial expression sequences and has practical potential for controllable avatars, with future work aiming to expand expression vocabulary via LLMs and improve realism in challenging poses and occlusions.

Abstract

A 3D avatar typically has one of six cardinal facial expressions. To simulate realistic emotional variation, we should be able to render a facial transition between two arbitrary expressions. This study presents a new framework for instruction-driven facial expression generation that produces a 3D face and, starting from an image of the face, transforms the facial expression from one designated facial expression to another. The Instruction-driven Facial Expression Decomposer (IFED) module is introduced to facilitate multimodal data learning and capture the correlation between textual descriptions and facial expression features. Subsequently, we propose the Instruction to Facial Expression Transition (I2FET) method, which leverages IFED and a vertex reconstruction loss function to refine the semantic comprehension of latent vectors, thus generating a facial expression sequence according to the given instruction. Lastly, we present the Facial Expression Transition model to generate smooth transitions between facial expressions. Extensive evaluation suggests that the proposed model outperforms state-of-the-art methods on the CK+ and CelebV-HQ datasets. The results show that our framework can generate facial expression trajectories according to text instruction. Considering that text prompts allow us to make diverse descriptions of human emotional states, the repertoire of facial expressions and the transitions between them can be expanded greatly. We expect our framework to find various practical applications More information about our project can be found at https://vohoanganh.github.io/tg3dfet/

Figures (18)

• Figure 1: Illustration of our framework wherein it accepts a text instruction with a face image (source) as input and generates a 3D face from it. Then, it transforms the facial expression, from disgust (expression 1) to happiness (expression 2) via a neutral expression.
• Figure 2: Overview of the proposed framework consisting of two major modules: Facial Expression Transition (FET) and Face Rendering (FR). First, textual vector $x_t$ is obtained from the pre-trained CLIP encoder for the FET module. Simultaneously, latent representations $z_{p}$ and $z_e$ are drawn from $\mathcal{N}(0, I)$. Then, the latent vectors are processed and concatenated to obtain $\hat{x}^f$. Afterward, the IFED module utilizes $x^t$ and $\hat{x}^f$ as inputs to create conditional feature vectors $\hat{x}_{emb}^e$, and $\hat{x}_{emb}^{p}$ for the pose and expression decoders. Subsequently, for smoothness of the facial expression sequence, the facial expression in the source image $(e_s, \theta_s)$ is subjected to linear interpolation and the specific facial expressions $(e_0, \theta_0), (e_1, \theta_1)$, generated by the I2FET decoders. These sequences are then combined with the shape $\phi_s$ and camera $c_s$ parameters of the source image, obtained from DECA, to form facial expression trajectories $\{s_s^{(1)},s^{(i)},s^{(k)}_{e_0},s^{(j)}, s_{e_1}^{(T)}\}$. For FR, head mesh reconstruction produces a flame mesh sequence $\{m_s^{(1)},m^{(i)},m^{(k)}_{e_0},m^{(j)}, m_{e_1}^{(T)}\}$ aligned with the facial expression trajectories and the texture encoder extracts the facial appearance from the source image. Finally, the rendering module generates the facial appearances $\{y_s^{(1)},y^{(i)},y^{(k)}_{e_0},y^{(j)}, y_{e_1}^{(T)}\}$ within the facial expression trajectories using the flame mesh sequence and the extracted facial appearance.
• Figure 3: Flow diagram of IFED (Instruction-Driven Facial Expression Decomposer). It consists of CAFT and the decomposition of parameters. The major function of CAFT is to introduce cross-attention between the facial parameters and text instruction. The two outputs from CAFT pass through layer normalization and are combined to create the fused feature vector $x^{fused}$. Then, pose projection function $\mathcal{P}_{p}$ and expression projection function $\mathcal{P}_{e}$ are used to decompose $x^{fused}$ into two conditional vectors for facial expression ($x_{emb}^e$) and pose ($x_{emb}^{p}$).
• Figure 4: Detailed architecture of the I2FET (Instruction to Facial Expression Transition) method. Given an instruction and a sequence of facial parameters, including pose and expression, the IFED module utilizes the facial parameters and textual description extracted by CLIP to produce the corresponding conditional feature vectors for pose $x_{emb}^{p}$ and expression $x_{emb}^e$, respectively. The pose and expression encoders then map this sequence of facial parameters and conditional feature vectors to latent representations corresponding to pose and expression. These latent vectors, along with the textual description, are passed through the IFED module to generate the conditional feature vectors $\hat{x}_{emb}^e$ and $\hat{x}_{emb}^{p}$. Following this, the expression and pose decoders reconstruct expression $\hat{e}$ and pose $\hat{\theta}$ parameters using these latent representations and the conditional feature vectors $\hat{x}_{emb}^e$ and $\hat{x}_{emb}^{p}$, respectively. Finally, the pre-trained FLAME decoder reconstructs the vertex coordinates $\hat{v}$ based on $\hat{e}$ and $\hat{\theta}$.
• Figure 5: Statistics of the facial expressions utilized in this study for the CK+ (a) and CelebV-HQ (b) datasets, respectively, and a collection of sampled words extracted from the text instructions (c).