Point2Mesh: A Self-Prior for Deformable Meshes

TL;DR

Point2Mesh introduces a self-prior for reconstructing watertight meshes from point clouds by optimizing a CNN-inspired, weight-sharing network that deforms a starting mesh to shrink-wrap the input. The self-prior leverages global self-similarity learned from the input shape, enabling robust reconstruction under noise, missing data, and unoriented normals, without requiring training data. The method combines edge-based CNNs (MeshCNN) with a beam-gap loss and a coarse-to-fine optimization, maintaining mesh connectivity and genus while using a differentiable sampling and Chamfer distance to align to the point cloud. Experimental results on real scans and benchmarks show superior performance to Poisson, DGP, and PCN in denoising and completion, and ablations demonstrate the importance of the self-prior.

Abstract

In this paper, we introduce Point2Mesh, a technique for reconstructing a surface mesh from an input point cloud. Instead of explicitly specifying a prior that encodes the expected shape properties, the prior is defined automatically using the input point cloud, which we refer to as a self-prior. The self-prior encapsulates reoccurring geometric repetitions from a single shape within the weights of a deep neural network. We optimize the network weights to deform an initial mesh to shrink-wrap a single input point cloud. This explicitly considers the entire reconstructed shape, since shared local kernels are calculated to fit the overall object. The convolutional kernels are optimized globally across the entire shape, which inherently encourages local-scale geometric self-similarity across the shape surface. We show that shrink-wrapping a point cloud with a self-prior converges to a desirable solution; compared to a prescribed smoothness prior, which often becomes trapped in undesirable local minima. While the performance of traditional reconstruction approaches degrades in non-ideal conditions that are often present in real world scanning, i.e., unoriented normals, noise and missing (low density) parts, Point2Mesh is robust to non-ideal conditions. We demonstrate the performance of Point2Mesh on a large variety of shapes with varying complexity.

Figures (16)

• Figure 1: Reconstructing a complete mesh from a point cloud with missing regions using a smooth-prior ignores the character of the global shape. The self-prior in Point2Mesh inherently learns and leverages the recurrences present within a single shape, leading to a more plausible reconstruction.
• Figure 2: The CNN structure inherently prefers reconstructing natural shapes. Graph of reconstruction error vs. optimization iterations (left) for reconstructing four different shapes: (a) natural shape, (b) shape + noise, (c) shuffled vertices and (d) uniform noise. The network is able to best reconstruct the natural shape (a), and struggles to reconstruct noisy and chaotic (unnatural) shapes (b,c,d).
• Figure 3: Overview of Point2Mesh framework in a level $l$. The initial mesh $M_{l-1}$ and fixed random constant $C_{l-1}$ are input to the network (self-prior), which outputs a differential displacement vector per edge $\Delta \hat{E}_l$. The differential displacement per vertex $\Delta \hat{V}_l$ is calculated by averaging the displacements for each of its incident edges. The reconstructed mesh $\hat{M}_l$ has vertices given by the vertices of $\hat{M}_l$ plus $\Delta \hat{V}_l$, and the connectivity of $\hat{M}_l$. The reconstructed mesh is sampled to get $\hat{Y}_l$ which is compared against the input point cloud $X$. This loss is back-propagated in order to update self-prior network weights.
• Figure 4: Illustration of edge displacements predicted by network (left) and then averaged (right).
• Figure 5: The input point cloud is sampled from a (ground-truth) mesh, with added noise and missing regions. A smooth-prior reconstructs the surface locally, oblivious to the global shape. While the self-prior retains the reoccurring ridges in the back of the ankylosaurus and it smooths bumps which originated from noise. Note the smooth reconstruction along the tail and side of the body.