DiffLocks: Generating 3D Hair from a Single Image using Diffusion Models

TL;DR

DiffLocks tackles the challenge of generating realistic 3D hair from a single image by constructing the largest synthetic 3D hair dataset to date and training an image-conditioned diffusion model that decodes directly into 3D hair strands. It replaces prior low-dimensional scalp representations with a dense scalp texture of latent strand codes and a probabilistic density map, enabling detailed curls and balding patterns without post-processing. The approach leverages HDiT diffusion with DINOv2 image conditioning and cross-attention, producing high-fidelity strands that transfer to real imagery and real-time engines like Unreal through an Alembic export. Across extensive qualitative and quantitative evaluations, DiffLocks demonstrates state-of-the-art performance, particularly for curly and afro-like hairstyles, while maintaining fast, integration-friendly runtime performance.

Abstract

We address the task of generating 3D hair geometry from a single image, which is challenging due to the diversity of hairstyles and the lack of paired image-to-3D hair data. Previous methods are primarily trained on synthetic data and cope with the limited amount of such data by using low-dimensional intermediate representations, such as guide strands and scalp-level embeddings, that require post-processing to decode, upsample, and add realism. These approaches fail to reconstruct detailed hair, struggle with curly hair, or are limited to handling only a few hairstyles. To overcome these limitations, we propose DiffLocks, a novel framework that enables detailed reconstruction of a wide variety of hairstyles directly from a single image. First, we address the lack of 3D hair data by automating the creation of the largest synthetic hair dataset to date, containing 40K hairstyles. Second, we leverage the synthetic hair dataset to learn an image-conditioned diffusion-transfomer model that generates accurate 3D strands from a single frontal image. By using a pretrained image backbone, our method generalizes to in-the-wild images despite being trained only on synthetic data. Our diffusion model predicts a scalp texture map in which any point in the map contains the latent code for an individual hair strand. These codes are directly decoded to 3D strands without post-processing techniques. Representing individual strands, instead of guide strands, enables the transformer to model the detailed spatial structure of complex hairstyles. With this, DiffLocks can recover highly curled hair, like afro hairstyles, from a single image for the first time. Data and code is available at https://radualexandru.github.io/difflocks/

Figures (21)

• Figure 1: Given a single RGB image DiffLocks generates accurate 3D strands using a diffusion model. The model is trained on a novel synthetic hair dataset containing RGB images and corresponding 3D strands.
• Figure 1: Quantitative comparison with wu2022neuralhdhairzheng2023hairstep on yuksel2009hair. It includes straight and curly hairstyles, please refer to the SupMat for more visualizations.
• Figure 2: Method. Given a single RGB image, we use a pretrained DINOv2 model to extract local and global features, which are used to guide a scalp diffusion model. The scalp diffusion model denoises a density map and a scalp texture containing latent codes for strand geometry. Finally, we probabilistically sample texels from the scalp texture and decode the latent code $\mathbf{z}$ into strands of 256 points. Decoding in parallel 100.0K strands yields the final hairstyle.
• Figure 2: Quantitative comparison with wu2022neuralhdhairzheng2023hairstep on the DiffLocks evaluation set. Our results show substantial improvement over the baseline, particularly when reconstructing the backside of the hair due to our model's ability to learn a powerful prior over hairstyles.
• Figure 3: Synthetic data. Sample RGB images from our synthetic hair dataset. Each sample from the dataset contains an image at $768 \times 768$ resolution together with the corresponding 3D strand geometry of $\approx 100$K strands.