TurboPortrait3D: Single-step diffusion-based fast portrait novel-view synthesis

TL;DR

TurboPortrait3D tackles the challenge of fast, 3D-consistent portrait novel-view synthesis from a single image. It combines a feed-forward avatar generator (GP-Avatar) with a single-step diffusion refinement that uses an attention-reshaping block and a variable-noise training regime to deliver high-fidelity, identity-preserving views under low latency. The approach is trained on synthetic and real portrait data with LoRA adapters for efficiency and demonstrates both quantitative superiority and qualitative realism over state-of-the-art baselines, including diffusion- and avatar-based methods. The resulting system enables practical applications in telepresence and live-content creation, with potential extensions to video and interactive control for dynamic digital humans.

Abstract

We introduce TurboPortrait3D: a method for low-latency novel-view synthesis of human portraits. Our approach builds on the observation that existing image-to-3D models for portrait generation, while capable of producing renderable 3D representations, are prone to visual artifacts, often lack of detail, and tend to fail at fully preserving the identity of the subject. On the other hand, image diffusion models excel at generating high-quality images, but besides being computationally expensive, are not grounded in 3D and thus are not directly capable of producing multi-view consistent outputs. In this work, we demonstrate that image-space diffusion models can be used to significantly enhance the quality of existing image-to-avatar methods, while maintaining 3D-awareness and running with low-latency. Our method takes a single frontal image of a subject as input, and applies a feedforward image-to-avatar generation pipeline to obtain an initial 3D representation and corresponding noisy renders. These noisy renders are then fed to a single-step diffusion model which is conditioned on input image(s), and is specifically trained to refine the renders in a multi-view consistent way. Moreover, we introduce a novel effective training strategy that includes pre-training on a large corpus of synthetic multi-view data, followed by fine-tuning on high-quality real images. We demonstrate that our approach both qualitatively and quantitatively outperforms current state-of-the-art for portrait novel-view synthesis, while being efficient in time.

Figures (7)

• Figure 1: Teaser -- From a single frontal reference image, TurboPortriait3D is able to generate 3D-aware novel portrait views. Unoptimized, our approach does this at about 120 ms, or roughly 8 frames per second.
• Figure 2: TurboPortrait3D pipeline. From a single reference image, GP-Avatar chu2024gpavatar generates 3D-consistent novel views from predicted target viewpoints. These, along with the reference image, are encoded into diffusion latents. Noise is added only to the novel views, and a Stable Diffusion Turbo U-Net with our attention-reshaping block refines them in a single step. The reference image is discarded, and the refined views are supervised with corresponding ground-truth images via a discriminator. Our design enables feed-forward, single-step, 3D-aware portrait refinement with high identity fidelity.
• Figure 3: Attention-reshaping block. We reshape latents before and after the ResBlock and attention layers so identity and texture cues from the reference image propagate globally across all views, as introduced in Difix3D. This design achieves cross-view consistency without adding an extra U-Net for cross-attention, keeping the refinement lightweight and memory-efficient.
• Figure 4: Ava-256 validation results -- We report the qualitative results comparing state-of-the-art methods for 3D consistent, novel view portrait synthesis on Ava-256 validation data.
• Figure 5: NeRSemble validation results -- We report the qualitative results comparing state-of-the-art methods for 3D consistent, novel view portrait synthesis on NeRSemble validation data.