Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Capture, Canonicalize, Splat: Zero-Shot 3D Gaussian Avatars from Unstructured Phone Images

Emanuel Garbin, Guy Adam, Oded Krams, Zohar Barzelay, Eran Guendelman, Michael Schwarz, Matteo Presutto, Moran Vatelmacher, Yigal Shenkman, Eli Peker, Itai Druker, Uri Patish, Yoav Blum, Max Bluvstein, Junxuan Li, Rawal Khirodkar, Shunsuke Saito

TL;DR

This paper presents a novel, zero-shot pipeline for creating hyperrealistic, identity-preserving 3D avatars from a few unstructured phone images, featuring a generative canonicalization module that processes multiple unstructured views into a standardized, consistent representation.

Abstract

We present a novel, zero-shot pipeline for creating hyperrealistic, identity-preserving 3D avatars from a few unstructured phone images. Existing methods face several challenges: single-view approaches suffer from geometric inconsistencies and hallucinations, degrading identity preservation, while models trained on synthetic data fail to capture high-frequency details like skin wrinkles and fine hair, limiting realism. Our method introduces two key contributions: (1) a generative canonicalization module that processes multiple unstructured views into a standardized, consistent representation, and (2) a transformer-based model trained on a new, large-scale dataset of high-fidelity Gaussian splatting avatars derived from dome captures of real people. This "Capture, Canonicalize, Splat" pipeline produces static quarter-body avatars with compelling realism and robust identity preservation from unstructured photos.

Capture, Canonicalize, Splat: Zero-Shot 3D Gaussian Avatars from Unstructured Phone Images

TL;DR

This paper presents a novel, zero-shot pipeline for creating hyperrealistic, identity-preserving 3D avatars from a few unstructured phone images, featuring a generative canonicalization module that processes multiple unstructured views into a standardized, consistent representation.

Abstract

We present a novel, zero-shot pipeline for creating hyperrealistic, identity-preserving 3D avatars from a few unstructured phone images. Existing methods face several challenges: single-view approaches suffer from geometric inconsistencies and hallucinations, degrading identity preservation, while models trained on synthetic data fail to capture high-frequency details like skin wrinkles and fine hair, limiting realism. Our method introduces two key contributions: (1) a generative canonicalization module that processes multiple unstructured views into a standardized, consistent representation, and (2) a transformer-based model trained on a new, large-scale dataset of high-fidelity Gaussian splatting avatars derived from dome captures of real people. This "Capture, Canonicalize, Splat" pipeline produces static quarter-body avatars with compelling realism and robust identity preservation from unstructured photos.
Paper Structure (9 sections, 1 equation, 4 figures, 1 table)

This paper contains 9 sections, 1 equation, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Single-image 3D human reconstruction.
  4. Multi-view 3D human reconstruction.
  5. Generative models for 3D avatars.
  6. Method
  7. A high-fidelity human avatar dataset
  8. Generative view canonicalization
  9. Multi-view to 3D Gaussian splatting reconstruction

Figures (4)

  • Figure 1: Our "Capture, Canonicalize, Splat" pipeline. From a few unstructured phone photos (left), our generative canonicalization module synthesizes a set of 3D-consistent, identity-preserving views with fixed camera parameters (middle). These views are then lifted by our transformer-based reconstruction model, trained on our high-fidelity dataset, into a hyperrealistic 3D Gaussian splatting avatar (right).
  • Figure 2: Our high-fidelity dataset creation workflow. (1) We start with calibrated multi-view images of real individuals from a dome capture setup. (2) We optimize a high-fidelity Gaussian avatar for each subject, which serves as our ground truth. (3) From this ground-truth 3D model, we render extensive training data, including canonical multi-view sets and simulated unstructured captures, to train our models.
  • Figure 3: Limitations of synthetic training data. A model trained solely on synthetic data such as RenderPeople fails to capture realism. Given a real input photo (left), its output (right) exhibits an identity shift and an overly smooth, stylized appearance.
  • Figure 4: Importance of multiple views for identity preservation. Reconstruction from a single view often fails to preserve identity, especially for unseen areas. Our multi-view approach produces a more faithful and consistent result.