SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement

TL;DR

SF3D tackles single-image 3D reconstruction by generating textured, UV-unwrapped meshes with disentangled lighting and material properties, enabling relighting and practical downstream use. It integrates a high-resolution Enhanced Transformer, Material Net, Light Net, DMTet-based mesh refinement, and a fast UV-unwrapping/export pipeline to deliver textured meshes in about 0.5 s. Across GSO and OmniObject3D, SF3D achieves state-of-the-art geometry while using far fewer triangles and providing coherent textures and materials, outperforming baselines in CD and F-score. The approach enables rapid creation of usable 3D assets for games, AR/VR, and e-commerce, with training strategies that stabilize material estimation and shading under unknown illumination.

Abstract

We present SF3D, a novel method for rapid and high-quality textured object mesh reconstruction from a single image in just 0.5 seconds. Unlike most existing approaches, SF3D is explicitly trained for mesh generation, incorporating a fast UV unwrapping technique that enables swift texture generation rather than relying on vertex colors. The method also learns to predict material parameters and normal maps to enhance the visual quality of the reconstructed 3D meshes. Furthermore, SF3D integrates a delighting step to effectively remove low-frequency illumination effects, ensuring that the reconstructed meshes can be easily used in novel illumination conditions. Experiments demonstrate the superior performance of SF3D over the existing techniques. Project page: https://stable-fast-3d.github.io

Figures (8)

• Figure 1: SF3D Improvements on Different Prevalent Issues. Here, we compare our results with TripoSR tochilkin2024triposr. The top shows the effect of light bake-in when relighting the asset. SF3D produces a more plausible relighting. By not using vertex colors, our method is capable of encoding finer details while also having a lower polygon count. Our vertex displacement enables estimating smooth shapes, which do not introduce stair-stepping artifacts from marching cubes. Lastly, our material property prediction allows us to express a variety of different surface types.
• Figure 2: SF3D Overview. SF3D improves on TripoSR by addressing the issues in Fig. \ref{['fig:issues']} with 5 novel modules: 1. An enhanced transfomer for higher resolution triplanes (top left); 2. Material estimation with Material Net (bottom left); 3. Explicit illumination estimation using Light Net (bottom right); 4. Smooth mesh extraction with the estimation of vertex offsets and normals (top right); and finally 5. an export pipeline with fast UV-unwrapping (right).
• Figure 3: Triplane Resolution Aliasing. We found that low-resolution triplanes struggle with high-frequency and high-contrast textures and produce grid-like aliasing artifacts. Our method increases the triplane size from $64^2$ to $384^2$ allowing our method to reproduce these textures with fewer artifacts.
• Figure 4: Export Pipeline. Our export process starts with the mesh and is followed by UV unwrapping, occupancy and world position baking, material querying, and UV island margins.
• Figure 5: UV Unwrapping. Our UV unwrapping technique uses projection mapping, allowing each face to independently select a projection, enabling easy parallelization. A naive approach could lead to the same UV coordinates being assigned to different vertices due to occlusions. We identify potential overlaps from occlusions on the 2D mapped surfaces and relocate them to different areas within the UV atlas. Any remaining areas are placed in the bottom right of the UV atlas. This method minimizes distortion and ensures most surfaces are preserved in a connected area.