Neural 3D Strokes: Creating Stylized 3D Scenes with Vectorized 3D Strokes

TL;DR

The approach draws inspiration from image-to-painting methods, simulating the progressive painting process of human artwork with vector strokes, and synthesizes 3D scenes with significant geomet-ric and aesthetic stylization while maintaining a consis-tent appearance across different views.

Abstract

We present Neural 3D Strokes, a novel technique to generate stylized images of a 3D scene at arbitrary novel views from multi-view 2D images. Different from existing methods which apply stylization to trained neural radiance fields at the voxel level, our approach draws inspiration from image-to-painting methods, simulating the progressive painting process of human artwork with vector strokes. We develop a palette of stylized 3D strokes from basic primitives and splines, and consider the 3D scene stylization task as a multi-view reconstruction process based on these 3D stroke primitives. Instead of directly searching for the parameters of these 3D strokes, which would be too costly, we introduce a differentiable renderer that allows optimizing stroke parameters using gradient descent, and propose a training scheme to alleviate the vanishing gradient issue. The extensive evaluation demonstrates that our approach effectively synthesizes 3D scenes with significant geometric and aesthetic stylization while maintaining a consistent appearance across different views. Our method can be further integrated with style loss and image-text contrastive models to extend its applications, including color transfer and text-driven 3D scene drawing. Results and code are available at http://buaavrcg.github.io/Neural3DStrokes.

Figures (11)

• Figure 1: We propose a method to stylize a 3D scene from multi-view 2D images using vectorized 3D strokes based on geometric primitives and splines. The four scenes from left to right are drawn with axis-aligned box, oriented box, ellipsoid, and cubic Bézier curve, respectively.
• Figure 2: Our method learns a vectorized stroke field instead of MLP-based implicit representation to represent a 3D scene.
• Figure 3: Differential region function by approximating step function with the CDF of Laplace distribution. The parameter $\delta$ controls the width of the transitional area with respect to the SDF of 3D strokes, where a larger $\delta$ leads to smoother gradient at the cost of making the shape boundary blurrier.
• Figure 4: Novel view synthesis results of various 3D strokes on the synthetic scenes. Each scene contains 500 strokes. Our vectorized stroke representation is able to recover the 3D scenes with high fidelity while maintaining a strong geometric style defined by the strokes.
• Figure 5: Results of different types of 3D strokes on the face-forwarding scenes. Each scene contains 1000 strokes.