3D Geometric Shape Assembly via Efficient Point Cloud Matching

TL;DR

This work introduces Proxy Match Transform (PMT), a low-complexity high-order feature transform that approximates traditional high-order convolutions for efficient point-cloud matching. By enforcing orthogonality constraints on proxy tensors and applying PMT in a coarse-to-fine PMTR pipeline, the approach localizes and refines mating-surface correspondences with sub-quadratic complexity $O( ext{max}(| ext{X}|,| ext{Y}|) imes D_{ ext{proxy}})$ where $D_{ ext{proxy}}\ll | ext{X}|,| ext{Y}|$. Experiments on the Breaking Bad dataset show PMTR achieving state-of-the-art results for pairwise and multi-part shape assembly, with clear memory and computation advantages over prior high-order methods, validated by ablations on proxy sharing and constraint losses. The method promises practical impact for 3D assembly tasks in robotics, CAD, and manufacturing, enabling accurate, scalable alignment of complex geometric parts.

Abstract

Learning to assemble geometric shapes into a larger target structure is a pivotal task in various practical applications. In this work, we tackle this problem by establishing local correspondences between point clouds of part shapes in both coarse- and fine-levels. To this end, we introduce Proxy Match Transform (PMT), an approximate high-order feature transform layer that enables reliable matching between mating surfaces of parts while incurring low costs in memory and computation. Building upon PMT, we introduce a new framework, dubbed Proxy Match TransformeR (PMTR), for the geometric assembly task. We evaluate the proposed PMTR on the large-scale 3D geometric shape assembly benchmark dataset of Breaking Bad and demonstrate its superior performance and efficiency compared to state-of-the-art methods. Project page: https://nahyuklee.github.io/pmtr.

Figures (6)

• Figure 1: Given a correlation score at position $(\mathbf{x}, \mathbf{y})$ (the edge between highlighted nodes) and its neighboring scores (all other edges), vanilla high-order feature transform (shown on the left) leads to quadratic complexity due to its demand for memory-intensive pairwise correlation scores. The product of two PMTs (shown on the right) effectively approximates this high-order transform only with sub-quadratic complexity by avoiding direct construction of correlation scores, instead exchanging information through a low-dimensional proxy tensor. The red/blue nodes and black edges represent the source/target features and the correlation scores between them, respectively.
• Figure 2: Overall pipeline of the Proxy Match TransformeR (PMTR) for pairwise shape assembly. The proposed architecture consists of coarse-level matching and fine-level matching. Each part of matching uses coarse-level features and fine-level features, respectively, acquired from the KPConv-FPN backbone as their input. Each matcher consists of $N_t$ PMT layers in series. See Sec. \ref{['sec:overall']} for details.
• Figure 3: (a) t-SNE visualization of proxy tensor (colored in purple), source features $\mathbf{F}_{\mathcal{X}_{c}}$ and target features $\mathbf{F}_{\mathcal{Y}_{c}}$. The source and target features are colored in warm (red) and cool (blue) tones, respectively, and those on mating surfaces are colored in orange and lightblue. (b) Feature visualization in 3D space. Source $\mathcal{X}_{1}$ and target features $\mathcal{Y}_{1}$ with closer proximity to the proxy tensor are highlighted in red and blue, respectively, and features on mating surfaces are highlighted in orange and lightblue. For this visualization, we use a proxy tensor at a head index of $h=0$: $\mathbf{P}^{(0)}$.
• Figure 4: Qualitative results of pairwise shape assembly (Upper row) and multipart shape assembly (Bottom row) on Breaking Bad dataset.
• Figure 5: Additional qualitative results of pairwise shape assembly on Breaking Bad dataset.