PatchAlign3D: Local Feature Alignment for Dense 3D Shape understanding

TL;DR

PatchAlign3D addresses the gap between global 3D foundation models and fine-grained local reasoning by learning language-aligned patch-level features directly from point clouds. It introduces a two-stage pre-training workflow: Stage 1 distills dense 2D priors into 3D patch tokens, and Stage 2 aligns these tokens with text through a multi-positive contrastive objective, enabling zero-shot 3D part segmentation without test-time rendering. The method achieves state-of-the-art results across five benchmarks, with sharp segmentation boundaries and fast inference, highlighting its potential for open-world 3D understanding. This approach significantly reduces error from noisy annotations and prompt engineering, providing a practical, efficient path toward real-time 3D part reasoning in diverse applications.

Abstract

Current foundation models for 3D shapes excel at global tasks (retrieval, classification) but transfer poorly to local part-level reasoning. Recent approaches leverage vision and language foundation models to directly solve dense tasks through multi-view renderings and text queries. While promising, these pipelines require expensive inference over multiple renderings, depend heavily on large language-model (LLM) prompt engineering for captions, and fail to exploit the inherent 3D geometry of shapes. We address this gap by introducing an encoder-only 3D model that produces language-aligned patch-level features directly from point clouds. Our pre-training approach builds on existing data engines that generate part-annotated 3D shapes by pairing multi-view SAM regions with VLM captioning. Using this data, we train a point cloud transformer encoder in two stages: (1) distillation of dense 2D features from visual encoders such as DINOv2 into 3D patches, and (2) alignment of these patch embeddings with part-level text embeddings through a multi-positive contrastive objective. Our 3D encoder achieves zero-shot 3D part segmentation with fast single-pass inference without any test-time multi-view rendering, while significantly outperforming previous rendering-based and feed-forward approaches across several 3D part segmentation benchmarks. Project website: https://souhail-hadgi.github.io/patchalign3dsite/

Figures (7)

• Figure 1: PatchAlign3D pre-training. Given an input point cloud, we extract multi-view visual features using a 2D backbone and back-project them into 3D space. In Stage 1, the 3D transformer encoder operates on sampled point cloud patches and learns to align its output patch tokens with the back-projected visual features. In Stage 2, we initialize from Stage 1, freeze all earlier layers, and train only the last transformer block and projector to align patch-level features with textual embeddings in a contrastive manner.
• Figure 2: Qualitative comparisons on ShapeNetPart yi2016scalable. We show ground truth (top row) and predictions from COPS garosi20253d, Find3D ma2025find, and PatchAlign3D (rows 2–4) across six representative shapes. The part legends below each column indicate the semantic labels used for zero-shot prediction. PatchAlign3D produces noticeably more precise and coherent segmentations, despite relying solely on an encoder and patch-level features.
• Figure 3: Qualitative comparison on non-rigid human shapes from FAUST bogo2014faustabdelreheem2023satr. We show ground truth and predictions from COPS garosi20253d, Find3D ma2025find, and PatchAlign3D across three representative shapes (columns). The part legend below specifies the semantic labels used for zero-shot prediction. PatchAlign3D produces cleaner segmentations than prior methods and is less noisy than Find3D's encoder–decoder outputs.
• Figure 4: Feature comparison across stages. We visualize features from DINOv2, Stage 1, and Stage 1 + Stage 2 of our approach on example point clouds from the validation split of the training data. Stage 1 refines DINOv2 features, and Stage 2 further preserves them while assigning downstream text capabilities.
• Figure 6: Text-to-feature similarity visualization. We compare PatchAlign3D to Find3D by visualizing similarities between a textual query (e.g., "ear", "nose") and the dense features on a validation point cloud. Yellow indicates higher similarity. PatchAlign3D produces sharper and more localized responses, while Find3D often shows diffuse signals with weaker semantic localization.