STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting

TL;DR

STAR tackles pose-bias and animation artifacts in text-driven 4D avatar generation by integrating in-network motion retargeting with skeleton-conditioned T2I/T2V priors and a hybrid SDS framework. The method progressively optimizes geometry, texture, and motion using occlusion-aware conditioning and frame-consistent supervision, guided by hierarchical regularization. Empirical results show state-of-the-art text-4D alignment and vivid, temporally coherent animations, validated by both quantitative metrics and human judgments. The work demonstrates a practical path toward end-to-end, text-to-4D avatar synthesis with improved stability and fidelity, while outlining concrete limitations and future directions for richer representations.

Abstract

The creation of 4D avatars (i.e., animated 3D avatars) from text description typically uses text-to-image (T2I) diffusion models to synthesize 3D avatars in the canonical space and subsequently applies animation with target motions. However, such an optimization-by-animation paradigm has several drawbacks. (1) For pose-agnostic optimization, the rendered images in canonical pose for naive Score Distillation Sampling (SDS) exhibit domain gap and cannot preserve view-consistency using only T2I priors, and (2) For post hoc animation, simply applying the source motions to target 3D avatars yields translation artifacts and misalignment. To address these issues, we propose Skeleton-aware Text-based 4D Avatar generation with in-network motion Retargeting (STAR). STAR considers the geometry and skeleton differences between the template mesh and target avatar, and corrects the mismatched source motion by resorting to the pretrained motion retargeting techniques. With the informatively retargeted and occlusion-aware skeleton, we embrace the skeleton-conditioned T2I and text-to-video (T2V) priors, and propose a hybrid SDS module to coherently provide multi-view and frame-consistent supervision signals. Hence, STAR can progressively optimize the geometry, texture, and motion in an end-to-end manner. The quantitative and qualitative experiments demonstrate our proposed STAR can synthesize high-quality 4D avatars with vivid animations that align well with the text description. Additional ablation studies shows the contributions of each component in STAR. The source code and demos are available at: \href{https://star-avatar.github.io}{https://star-avatar.github.io}.

Figures (6)

• Figure 1: Typical Optimization-by-Animation Pipeline (Top) v.s. Ours (Bottom) for Text-based 4D Avatar Generation.Top. Previous methods (e.g., TADA liao2024tada) first optimize the 3D representation in fixed canonical pose and subsequently apply motions for animation, which lack shape diversity and exhibit animation artifacts. Bottom. The proposed STAR for 4D avatar generation. It leverages in-network motion retargeting and hybrid SDS to jointly update geometry, texture, and motions to achieve vivid 4D avatar from only text description.
• Figure 2: Examples of Generated 4D Avatars. We propose STAR to produce high-fidelity 4D avatars from only text description. For each sample, we showcase: Left. Face and full-body of the textured 3D avatar and its normal map, and Right. Randomly selected frames of the 4D animation in two different camera views. Best viewed in color and zoom-in.
• Figure 3: Overview of the Proposed STAR.Left. Given a text description, we initialize the human motion with pretrained text-to-motion model zhang2024motiondiffuse. Note that the typical optimization-by-animation paradigm easily yields deteriorated body structures and animation artifacts for 4D avatar generation. Right. We eliminate the potential pose distribution bias in the SDS-based optimization by integrating the retargeted motion for animation. With the personalized and occlusion-aware skeleton, we leverage the hybrid T2I and T2V diffusion models to provide 3D consistent priors that progressively optimize the geometry, texture, and motion to produce 4D avatar in an end-to-end manner.
• Figure 4: Qualitative Comparisons of the Canonical 3D Avatar. For each method, we showcase examples of different prompts varying from real-world human to cartoon characters. The proposed STAR exhibits better text-geometry/texture alignment while ensuring shape variance and view consistency. As result, it presents visually better results compared to prior art.
• Figure 5: Qualitative Comparisons of 4D Avatars. For each sample, we showcase random selected frames of each avatar. The proposed STAR generates vivid 4D avatars which faithfully align with the given text description. The avatars generated by the comparison methods either show deteriorated body structures or animation artifacts.