Learning Self-Prior for Mesh Inpainting Using Self-Supervised Graph Convolutional Networks

TL;DR

This work addresses mesh inpainting for incomplete triangular meshes without relying on large training datasets, while preserving the original polygonal mesh format. It introduces self-supervised graph convolutional networks (SGCN and MGCN) that deform an oversmoothed hole-filled mesh via per-vertex displacements, trained with synthetic fake holes to provide supervision in non-hole regions using losses on vertex positions $E_{pos}$, facet normals $E_{nrm}$, and normal smoothing $E_{reg}$. The approach demonstrates state-of-the-art performance against traditional, dataset-free baselines and shows robustness to shapes rarely seen in datasets, with a refinement step further improving non-hole regions; code and data are public. This technique enables reliable, topology-preserving mesh inpainting without dataset prerequisites, offering a practical alternative to data-driven methods for challenging or uncommon geometries.

Abstract

In this paper, we present a self-prior-based mesh inpainting framework that requires only an incomplete mesh as input, without the need for any training datasets. Additionally, our method maintains the polygonal mesh format throughout the inpainting process without converting the shape format to an intermediate one, such as a voxel grid, a point cloud, or an implicit function, which are typically considered easier for deep neural networks to process. To achieve this goal, we introduce two graph convolutional networks (GCNs): single-resolution GCN (SGCN) and multi-resolution GCN (MGCN), both trained in a self-supervised manner. Our approach refines a watertight mesh obtained from the initial hole filling to generate a complete output mesh. Specifically, we train the GCNs to deform an oversmoothed version of the input mesh into the expected complete shape. The deformation is described by vertex displacements, and the GCNs are supervised to obtain accurate displacements at vertices in real holes. To this end, we specify several connected regions of the mesh as fake holes, thereby generating meshes with various sets of fake holes. The correct displacements of vertices are known in these fake holes, thus enabling training GCNs with loss functions that assess the accuracy of vertex displacements. We demonstrate that our method outperforms traditional dataset-independent approaches and exhibits greater robustness compared with other deep-learning-based methods for shapes that infrequently appear in shape datasets. Our code and test data are available at https://github.com/astaka-pe/SeMIGCN.

Figures (19)

• Figure 1: Overview of our self-supervised mesh inpainting. Our method, using only an input mesh with missing regions, consists of training and evaluation phases. In the training phase, the GCN is trained with hole-augmented meshes having different fake hole regions. The evaluation phase inputs the mesh $\mathcal{M}_{\text{in}}$ with only real holes and outputs the hole-completed mesh $\mathcal{M}_{\text{cmp}}$. Refinement, performed only in the evaluation phase, is then applied to improve the vertex positions at both hole and non-hole regions.
• Figure 2: Generation of initial mesh $\mathcal{M}_{\text{init}}$. We first obtain a watertight manifold mesh using MeshFix attene2010meshfix (inserted vertices are colored pink) and then apply triangular remeshing alliez2003remesh to achieve uniform vertex density.
• Figure 3: An oversmoothed mesh and meshes colored based on two geometric features. Local features are recognizable in the right mesh, colorized using vertex displacements, but not in the center, colored using vertex positions. This shows that local features can be extracted by subtracting vertex positions of the oversmoothed mesh from those of the original. The two color spheres in the center and right are identical but viewed from opposite sides to present the colors for outward and inward displacements.
• Figure 4: Meshes simplified using QEMs garland1997qem defined on a smoothed mesh. Each simplification reduces the number of vertices to $60%$, i.e., $N_{k} = \mathrm{round}(0.6 N_{k-1})$.
• Figure 5: Self-supervision of the proposed GCN, where the GCN predicts the displacement of each vertex to reproduce the inpainted shape from the oversmoothed shape. We train the GCN in a self-supervised manner to predict the vertex displacements by filling zeros to the displacements at real and fake holes (colored pink and orange, respectively).