Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Domain Agnostic Latent Embeddings of 3D Faces for Zero-shot Animal Expression Transfer

Yue Wang, Lawrence Amadi, Xiang Gao, Yazheng Chen, Yuanpeng Liu, Ning Lu, Xianfeng Gu

TL;DR

This work tackles zero-shot cross-species facial expression transfer by learning a mesh-agnostic latent embedding that disentangles identity and expression. It combines intrinsic features ($HKS$, $WKS$) with a DiffusionNet encoder and an attention-based fusion to predict a per-vertex Jacobian field that deforms animal meshes without animal supervision. The approach is trained exclusively on human expression data and demonstrates plausibility in transferring expressions to animal identities, effectively narrowing the geometric gap between human and animal faces. Key contributions include the intrinsic-geometry representation, vertex-wise Jacobian deformation, and latent-space analysis that elucidates identity and expression encodings, enabling expressive non-human avatars for VR/AR and animation.

Abstract

We present a zero-shot framework for transferring human facial expressions to 3D animal face meshes. Our method combines intrinsic geometric descriptors (HKS/WKS) with a mesh-agnostic latent embedding that disentangles facial identity and expression. The ID latent space captures species-independent facial structure, while the expression latent space encodes deformation patterns that generalize across humans and animals. Trained only with human expression pairs, the model learns the embeddings, decoupling, and recoupling of cross-identity expressions, enabling expression transfer without requiring animal expression data. To enforce geometric consistency, we employ Jacobian loss together with vertex-position and Laplacian losses. Experiments show that our approach achieves plausible cross-species expression transfer, effectively narrowing the geometric gap between human and animal facial shapes.

Learning Domain Agnostic Latent Embeddings of 3D Faces for Zero-shot Animal Expression Transfer

TL;DR

This work tackles zero-shot cross-species facial expression transfer by learning a mesh-agnostic latent embedding that disentangles identity and expression. It combines intrinsic features (, ) with a DiffusionNet encoder and an attention-based fusion to predict a per-vertex Jacobian field that deforms animal meshes without animal supervision. The approach is trained exclusively on human expression data and demonstrates plausibility in transferring expressions to animal identities, effectively narrowing the geometric gap between human and animal faces. Key contributions include the intrinsic-geometry representation, vertex-wise Jacobian deformation, and latent-space analysis that elucidates identity and expression encodings, enabling expressive non-human avatars for VR/AR and animation.

Abstract

We present a zero-shot framework for transferring human facial expressions to 3D animal face meshes. Our method combines intrinsic geometric descriptors (HKS/WKS) with a mesh-agnostic latent embedding that disentangles facial identity and expression. The ID latent space captures species-independent facial structure, while the expression latent space encodes deformation patterns that generalize across humans and animals. Trained only with human expression pairs, the model learns the embeddings, decoupling, and recoupling of cross-identity expressions, enabling expression transfer without requiring animal expression data. To enforce geometric consistency, we employ Jacobian loss together with vertex-position and Laplacian losses. Experiments show that our approach achieves plausible cross-species expression transfer, effectively narrowing the geometric gap between human and animal facial shapes.
Paper Structure (24 sections, 9 equations, 8 figures)

This paper contains 24 sections, 9 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human-to-Human Expression Transfer
  4. Cross-Domain and Zero-Shot Expression Transfer
  5. Mesh-Agnostic and Intrinsic Deformation Methods
  6. Cross-Species Deformation
  7. Method
  8. Expression Transfer Framework
  9. Intrinsic Mesh Features.
  10. DiffusionNet Feature Encoding.
  11. Attention-Based Feature Fusion.
  12. Jacobian-Based Deformation Prediction.
  13. Training Strategy and Loss Functions.
  14. Latent Space Analysis and Visualization
  15. Dataset and Pretraining Setup.
  16. ...and 9 more sections

Figures (8)

  • Figure 1: Overview of the proposed framework. Top: Training. Per-vertex intrinsic features (HKS/WKS) extracted from identity and expression meshes are processed by DiffusionNet encoders, fused via cross-attention, and decoded into a per-vertex Jacobian field that defines local deformations. Training is supervised using human face triplets with both vertex position and Jacobian losses. Bottom: Inference. At test time, the same model transfers expressions from a human source to an unseen animal identity in a zero-shot manner, without requiring animal expression supervision or mesh pre-alignment.
  • Figure 2: t-SNE visualization of the identity latent space. Points are color-coded by identity, showing clear clustering across different expressions.
  • Figure 3: Top: Closest three identity pairs in the learned ID latent space. Bottom: Farthest three identity pairs, illustrating separation between distinct identities.
  • Figure 4: t-SNE visualization of the expression latent space, color-coded by shared expression category across identities.
  • Figure 5: Top: Closest three expression pairs in the expression latent space. Bottom: Farthest three expression pairs, highlighting semantic separation between distinct facial deformations.
  • ...and 3 more figures