Geometry Transfer for Stylizing Radiance Fields

TL;DR

Geometry Transfer introduces a depth-guided framework to stylize both geometry and appearance of radiance fields. It extracts a depth map as $\\mathcal{S}_{D}$ and uses a deformation grid $G_\Delta$ to transform geometry while keeping the density grid $G_\sigma$ fixed, enabling coherent color sampling from the original surface. Extending to RGB-D stylization, it employs geometry-aware nearest matching and patch-wise optimization with perspective augmentation to increase expressiveness. Experiments on LLFF and ScanNet show superior stylizations and favorable user preferences compared with prior 3D style transfer methods, and partial stylization is demonstrated via Panoptic Lifting. This work broadens 3D style transfer by explicitly modeling geometry, enabling more accurate and diverse stylizations.

Abstract

Shape and geometric patterns are essential in defining stylistic identity. However, current 3D style transfer methods predominantly focus on transferring colors and textures, often overlooking geometric aspects. In this paper, we introduce Geometry Transfer, a novel method that leverages geometric deformation for 3D style transfer. This technique employs depth maps to extract a style guide, subsequently applied to stylize the geometry of radiance fields. Moreover, we propose new techniques that utilize geometric cues from the 3D scene, thereby enhancing aesthetic expressiveness and more accurately reflecting intended styles. Our extensive experiments show that Geometry Transfer enables a broader and more expressive range of stylizations, thereby significantly expanding the scope of 3D style transfer.

Figures (9)

• Figure 1: Given a reference 3D scene and a pair of style guides: an RGB image and a depth map, we coherently stylize both the scene's appearance and shape to best express the given style.
• Figure 2: Overview of our method. First we pre-train TensoRF chen2022tensorf on real-world images to obtain the color grid $\mathcal{G}_c$ and density grid $\mathcal{G}_\sigma$, enabling photorealistic reconstruction. Subsequently, we extract VGG features from style images as an RGB-D pair to stylize the shape and appearance of radiance fields. Here, the shape is modified through the additional deformation grid $\mathcal{G}_\Delta$, while $\mathcal{G}_\sigma$ remains fixed.
• Figure 3: Comparisons of the stylized results obtained by optimizing the density grid (a), and by optimizing the deformation fields (b). When directly optimizing the density, background colors are assigned to the updated parts of the foreground object.
• Figure 4: Sampling w/ and w/o deformation fields. Comparisons of the sampling density $\sigma_i$ and color $c_i$ for a 3D point $\mathbf{x}_i$ with and without deformation fields. The curves represent the 2D projected surfaces of objects, where green depicts the stylized surface and blue the original surface. By sampling with deformation fields, we coherently sample both values from the original surface.
• Figure 5: Qualitative comparisons with SNeRF nguyen2022snerf, ARF zhang2022arf and Ref-NPR zhang2023refnpr on the trex and fern scenes mildenhall2019local.