3D Shape Completion with Test-Time Training

TL;DR

This work approaches the task of restoring incomplete shapes by separately predicting the fractured and newly restored parts, but ensuring these predictions are interconnected, using a decoder network motivated by related work on the prediction of signed distance functions (DeepSDF).

Abstract

This work addresses the problem of \textit{shape completion}, i.e., the task of restoring incomplete shapes by predicting their missing parts. While previous works have often predicted the fractured and restored shape in one step, we approach the task by separately predicting the fractured and newly restored parts, but ensuring these predictions are interconnected. We use a decoder network motivated by related work on the prediction of signed distance functions (DeepSDF). In particular, our representation allows us to consider test-time-training, i.e., finetuning network parameters to match the given incomplete shape more accurately during inference. While previous works often have difficulties with artifacts around the fracture boundary, we demonstrate that our overfitting to the fractured parts leads to significant improvements in the restoration of eight different shape categories of the ShapeNet data set in terms of their chamfer distances.

Figures (6)

• Figure 1: Overview about our method. While DeepMend gets a good rough estimation for the fractured (upper left) and restoration shape (upper right), we get via test-time training much sharper and more detailed shapes, especially w.r.t. the break surface. This results in restoration shapes that fit much better to the original input fractured shape (bottom). In particular, 3D printing as an application case will benefit from our approach.
• Figure 2: Visualization of the problem setting: Given a fractured shape $F$ (a), we want to predict the missing restoration shape $R$ (b) such that we get the complete shape $C=F\cup R$ (c).
• Figure 3: Network architecture of deepmend2022. The architecture is separated into two parts which predict the occupancy of the complete shape $o_C$ and the break set $o_B$ respectively. The input for both parts is the point coordinate $x\in\mathbb{R}^3$ and a latent code describing the geometry. Via \ref{['eq:oF']} and \ref{['eq:oR']} we can calculate the occupancies of the fractured shape $o_F$ and the restoration shape $o_R.$ We define the skip connection with $\oplus$ and the multiplication of the outputs of the two networks with $\odot$.
• Figure 4: Pipeline of our method with test-time training. After only optimizing the latent code to get a rough prediction of the restoration shape (blue, this is the pipeline of DeepMend deepmend2022), we use the predicted complete shape as well as the input fractured shape to finetune all network parameters and therefore get a more detailed restoration shape (red, our addition).
• Figure 5: Qualitative examples of the different methods. Rows 1-3 and 6 are taken from Dataset 1, Rows 4-5 are from Dataset 2. Even though the Chamfer distance does not change much, the ability of our method to fit the fractured shape well, does visually a huge difference.