Ghost on the Shell: An Expressive Representation of General 3D Shapes

TL;DR

Ghost-on-the-Shell (G-Shell) presents a unified, differentiable mesh representation that can model both watertight and non-watertight geometries by placing open surfaces on a learnable watertight template via a manifold Signed Distance Field ($mSDF$). The method enables efficient, rasterization-based inverse rendering and diffusion-based generative modeling for general 3D shapes, addressing topological flexibility beyond traditional watertight meshes. Key contributions include (i) a grid-based, Marching-Cubes-like extraction that jointly handles $SDF$ and $mSDF$ signs, (ii) regularization strategies for hole opening and topology control from limited views, and (iii) an extension of MeshDiffusion to generate non-watertight meshes (G-MeshDiffusion) on the G-Shell representation. Empirical results demonstrate state-of-the-art performance on non-watertight reconstruction and generation, faster training/inference compared to baselines, and effective handling of complex lighting and materials, thereby broadening the practical utility of 3D reconstruction and synthesis in realistic scenarios.

Abstract

The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they 1) enable fast physics-based rendering with realistic material and lighting, 2) support physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range of object shapes, any 3D representation must be able to model solid, watertight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight surfaces, we parameterize open surfaces by defining a manifold signed distance field on watertight templates. With this parameterization, we further develop a grid-based and differentiable representation that parameterizes both watertight and non-watertight meshes of arbitrary topology. Our new representation, called Ghost-on-the-Shell (G-Shell), enables two important applications: differentiable rasterization-based reconstruction from multiview images and generative modelling of non-watertight meshes. We empirically demonstrate that G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks, while also performing effectively for watertight meshes.

Figures (19)

• Figure 1: Left: Illustration of mesh extraction with G-Shell through a manifold signed distance on a surface; Right: Applications of G-Shell, multiview mesh reconstruction (top) and mesh generation (bottom).
• Figure 2: Look-up table for Marching Tetrahedra (up to rotation symmetry). Grid vertices with and without a red dot possess SDF values of opposite signs.
• Figure 3: Illustration of non-watertight mesh extraction from some watertight triangular mesh. $p_1, p_2$ are the positions of (watertight) mesh vertices. $\Delta p = \left\lVert p_1 - p_2\right\rVert$ and $\nu_1 > 0 > \nu_2$ are the corresponding mSDF values. The orange triangle is extracted and the blue polygon is discarded.
• Figure 4: G-Shell look-up table (up to rotational symmetry) for tetrahedral grids. Grid vertices with and without a red dot possess SDF values of opposite signs, and green dots on watertight mesh vertices indicates negative mSDF values. The pink regions represent the final extracted faces while the blue ones are the discarded regions on the watertight template mesh. Colored polygons other than triangles are cut along dashed lines.
• Figure 5: Comparison between reconstruction w/ G-Shell and other baseline methods on multiview reconstruction on DeepFashion3D dataset. Top row: reconstructed texture. Bottom 3 rows: reconstructed meshes.

Theorems & Definitions (3)

• Theorem 1: richards1963classification
• Theorem 2: Classification Theorem of Surfaces lee2010introduction
• Theorem 3: ahlfors2015riemann