3D Stylization via Large Reconstruction Model

TL;DR

The paper tackles 3D appearance stylization for single-image to 3D generation pipelines using large reconstruction models. It demonstrates that appearance-related information concentrates in late-stage attention blocks and introduces a training-free method that injects style image features into these blocks during cross-attention in InstantMesh. The approach balances geometry and style by restricting stylization to the last 4 transformer blocks and blending outputs with a fixed alpha, achieving superior style fidelity with fast runtimes. Results on diverse objects show strong style transfer while preserving 3D geometry, outperforming baselines in user studies and robustness.

Abstract

With the growing success of text or image guided 3D generators, users demand more control over the generation process, appearance stylization being one of them. Given a reference image, this requires adapting the appearance of a generated 3D asset to reflect the visual style of the reference while maintaining visual consistency from multiple viewpoints. To tackle this problem, we draw inspiration from the success of 2D stylization methods that leverage the attention mechanisms in large image generation models to capture and transfer visual style. In particular, we probe if large reconstruction models, commonly used in the context of 3D generation, has a similar capability. We discover that the certain attention blocks in these models capture the appearance specific features. By injecting features from a visual style image to such blocks, we develop a simple yet effective 3D appearance stylization method. Our method does not require training or test time optimization. Through both quantitative and qualitative evaluations, we demonstrate that our approach achieves superior results in terms of 3D appearance stylization, significantly improving efficiency while maintaining high-quality visual outcomes.

Figures (9)

• Figure 1: InstantMesh provides a two stage pipeline for single-image based reconstruction. First a multi-view diffusion model is used to generate images of the object from multiple views. Embedding of such views are then provided to a triplane decoder to generate 3D object.
• Figure 2: Overview of our method. We build upon InstantMesh which takes an input image of an object and generates 6 sparse view outputs, which are then processed by a ViT encoder. A transformer based reconstructor generates a triplane where cross attention is performed between triplane and image tokens. Given a style image, we also encode it with the ViT encoder. In the latter transformer blocks, we perform cross-attention with respect to the style image tokens and blend it with the original cross attention output. This training and optimization free approach results in high-quality 3D visual stylization.
• Figure 3: Qualitative results for using style image ViT-encoded embeddings in the cross-attention layers across the last 1, 2, 4, 6, 8, 10, and all 16 layers of the triplane decoder. We use the last 4 layers in our experiments which provide a good balance between visual stylization and content fidelity.
• Figure 4: Our method blends the cross attention output with respect to original and style images based on a blending factor $\alpha$. We set $\alpha=0.8$ in our experiments.
• Figure 5: Visual results of our method and competing methods.