Freeplane: Unlocking Free Lunch in Triplane-Based Sparse-View Reconstruction Models

TL;DR

Freeplane addresses the challenge of artifacts caused by inconsistent multi-view images in triplane-based feed-forward 3D reconstruction. It introduces frequency modulation of the triplane features and a strategy to combine triplanes before and after filtering, achieving higher quality textured meshes without any additional training or cost. The method, compatible with CRM and InstantMesh, yields smoother geometry and preserved textures, validated through qualitative visuals and quantitative metrics on standard benchmarks. Overall, Freeplane provides a practical, plug-and-play improvement to existing triplane priors, highlighting the importance of exploiting frequency characteristics in neural-field representations for robust 3D generation.

Abstract

Creating 3D assets from single-view images is a complex task that demands a deep understanding of the world. Recently, feed-forward 3D generative models have made significant progress by training large reconstruction models on extensive 3D datasets, with triplanes being the preferred 3D geometry representation. However, effectively utilizing the geometric priors of triplanes, while minimizing artifacts caused by generated inconsistent multi-view images, remains a challenge. In this work, we present \textbf{Fre}quency modulat\textbf{e}d tri\textbf{plane} (\textbf{Freeplane}), a simple yet effective method to improve the generation quality of feed-forward models without additional training. We first analyze the role of triplanes in feed-forward methods and find that the inconsistent multi-view images introduce high-frequency artifacts on triplanes, leading to low-quality 3D meshes. Based on this observation, we propose strategically filtering triplane features and combining triplanes before and after filtering to produce high-quality textured meshes. These techniques incur no additional cost and can be seamlessly integrated into pre-trained feed-forward models to enhance their robustness against the inconsistency of generated multi-view images. Both qualitative and quantitative results demonstrate that our method improves the performance of feed-forward models by simply modulating triplanes. All you need is to modulate the triplanes during inference.

Figures (9)

• Figure 1: We present Freeplane, a method that substantially refines the mesh quality of feed-forward generative models without additional costs: no training or fine-tuning, no extra memory required, and only a few lines of code.
• Figure 2: Our key observation is that inconsistent multi-view images cause high-frequency artifacts on the triplanes, resulting in low-quality meshes. Given ground-truth images, CRM can generate a smooth and highly detailed mesh. However, during the inference stage, using the inconsistent multi-view images from diffusion models causes apparent artifacts on the triplanes, leading to low-quality 3D assets.
• Figure 3: Freeplane Framework.(a) Feed-forward Pipeline. A single image is input into the multi-view diffusion model to generate six-view images, which are then fed into the triplane decoder. By querying the triplane features, Flexicubes are extracted to produce the textured mesh. We adopt the Freeplane approach on the triplanes. (b) Freeplane Operation. Low-frequency filtering is applied to modulate the original triplanes. Triplanes before filtering are used to compute texture-related features, while those after filtering are utilized to predict mesh geometry.
• Figure 4: We present triplanes and meshes w/o. and w. Freeplane from InstantMesh. Our approach erases the noises and produces apparent boundaries on the triplanes, leading to a smooth surface with geometry details.
• Figure 5: Qualitative Results: With Freeplane, both CRM and InstantMesh are able to generate smoother meshes with fewer artifacts while preserving their textures.