Learning Local Pattern Modularization for Point Cloud Reconstruction from Unseen Classes

TL;DR

The paper tackles the challenge of reconstructing 3D point clouds for unseen classes from single images by introducing a local, class-agnostic prior learned as regional patterns. It maps an input image to an initial shape $S$, decomposes $S$ into regions $R_m$, and uses $N$ region patterns $P_n$ to form pattern-modularized regions $R_m'$, which are further customized by a learned shift $t_m$ conditioned on the image latent $f_I$ to produce the final $F$. The approach, trained with a region-focused loss $L_{Region}$ and a shape loss $L_{Shape}$, achieves state-of-the-art reconstruction accuracy on unseen classes across ShapeNet and Pixel3D, while remaining interpretable through explicit region patterns. Ablation studies confirm the critical role of the local prior, pattern modularization, and customization, and demonstrate strong generalization with relatively few patterns and without camera poses or category priors. Overall, the method provides a practical and interpretable pathway to high-fidelity 3D reconstructions for unseen categories.

Abstract

It is challenging to reconstruct 3D point clouds in unseen classes from single 2D images. Instead of object-centered coordinate system, current methods generalized global priors learned in seen classes to reconstruct 3D shapes from unseen classes in viewer-centered coordinate system. However, the reconstruction accuracy and interpretability are still eager to get improved. To resolve this issue, we introduce to learn local pattern modularization for reconstructing 3D shapes in unseen classes, which achieves both good generalization ability and high reconstruction accuracy. Our insight is to learn a local prior which is class-agnostic and easy to generalize in object-centered coordinate system. Specifically, the local prior is learned via a process of learning and customizing local pattern modularization in seen classes. During this process, we first learn a set of patterns in local regions, which is the basis in the object-centered coordinate system to represent an arbitrary region on shapes across different classes. Then, we modularize each region on an initially reconstructed shape using the learned local patterns. Based on that, we customize the local pattern modularization using the input image by refining the reconstruction with more details. Our method enables to reconstruct high fidelity point clouds from unseen classes in object-centered coordinate system without requiring a large number of patterns or any additional information, such as segmentation supervision or camera poses. Our experimental results under widely used benchmarks show that our method achieves the state-of-the-art reconstruction accuracy for shapes from unseen classes. The code is available at https://github.com/chenchao15/Unseen.

Figures (11)

• Figure 1: We learn to reconstruct shapes from unseen classes by learning a local class-agnostic prior with region patterns, such as (a) 4 patterns, (b) 8 patterns, (c) 10 patterns, or (d) 16 patterns. Using region patterns, we modularize each region of the initial reconstruction. Based on that, we obtain our final reconstruction by customizing these pattern modularized regions.
• Figure 2: The demonstration of our method. We aim to reconstruct a point clouds $\bm{F}$ from input image $\bm{I}$, where $\bm{F}$ may come from classes that are not seen during training.
• Figure 3: The architecture of pattern modularizer.
• Figure 4: The architecture of modularization customizer.
• Figure 5: The visual comparison under seen classes in ShapeNet.