Advances in 3D Neural Stylization: A Survey

TL;DR

Advances in 3D Neural Stylization introduces a taxonomy for neural stylization across 3D representations, surveys mesh, neural field, volume, and point-cloud approaches, and provides a benchmark and datasets to guide future work. It explains how 2D NST principles extend to 3D through image/text guidance, diffusion priors, and 3D-aware priors, and reviews methods from mesh geometry/texture editing to NeRF-based view synthesis and editing. The survey offers practical guidelines on achieving 3D consistency, controllability, efficiency, and evaluation, and discusses open challenges in large-scale, multi-modal, and real-time stylization. It also highlights applications in 3D asset design, avatars, NPR, PBR, and industrial production, and provides resources to benchmark and compare future techniques.

Abstract

Modern artificial intelligence offers a novel and transformative approach to creating digital art across diverse styles and modalities like images, videos and 3D data, unleashing the power of creativity and revolutionizing the way that we perceive and interact with visual content. This paper reports on recent advances in stylized 3D asset creation and manipulation with the expressive power of neural networks. We establish a taxonomy for neural stylization, considering crucial design choices such as scene representation, guidance data, optimization strategies, and output styles. Building on such taxonomy, our survey first revisits the background of neural stylization on 2D images, and then presents in-depth discussions on recent neural stylization methods for 3D data, accompanied by a benchmark evaluating selected mesh and neural field stylization methods. Based on the insights gained from the survey, we highlight the practical significance, open challenges, future research, and potential impacts of neural stylization, which facilitates researchers and practitioners to navigate the rapidly evolving landscape of 3D content creation using modern artificial intelligence.

Figures (12)

• Figure 1: The survey delves into the realm of neural stylization on diverse 3D representations, including meshes, point clouds, volume, and neural fields. The neural stylization with visual, textual and geometric features retrieved from large-scale neural models empowers artistic, photorealistic, and semantic style transformation of the geometry and appearance of 3D scenes. Images adapted from liu2018paparazzima2023xcao2020psnetyin20213dstylenetwang2023tsnerfzhang2022arfzhang2023refsong2023blendinghaque2023instructnerf.
• Figure 2: A general overview of mesh-based and radiance field-based rendering pipelines. Images adapted from yin20213dstylenetkim2024fprfchen2024surveyavrahami2022blendedspline_2023.
• Figure 3: Structure of our survey.
• Figure 4: Pipeline comparisons of 2D neural style transfer. (a) Single-style transfer via optimization gatys2016imagekwon2022clipstylerjohnson2016perceptual. (b) Arbitrary style transfer via feature fusion or transformation huang2017arbitraryli2019learningliu2021adaattn. (c) Image-to-image translation with style condition via generative models huang2018multimodaldeng2022stytr2wen2023capzhang2023inversion.
• Figure 5: 3D generation architecture with score distillation sampling loss. A pre-trained denoising U-Net supervises NeRF optimization. Image adapted from poole2022dreamfusion.

Theorems & Definitions (1)

• Definition 1