POMA-3D: The Point Map Way to 3D Scene Understanding

TL;DR

POMA-3D addresses the limited pretrained priors for 3D scene understanding by introducing point maps as a bridge between 2D foundation-model priors and 3D geometry. The method pretrains a 3D encoder on the ScenePoint dataset using two objectives: view-to-scene vision–language alignment to learn CLIP-aligned embeddings and POMA-JEPA to enforce multi-view geometric consistency, in a two-stage regime that warms up with 2D-derived data before leveraging multi-view room scenes. ScenePoint combines 6.5K real indoor scenes with 1M image-scene captions, enabling large-scale pretraining with pseudo point maps generated from 2D images via VGGT, and achieves strong zero-shot and finetuned performance across 3D QA, embodied navigation, scene retrieval, and embodied localization using only geometric inputs. The results demonstrate that transferring rich 2D priors through point maps substantially mitigates data scarcity in 3D learning and provides a scalable path toward generalizable 3D understanding for both specialist tasks and generalist systems.

Abstract

In this paper, we introduce POMA-3D, the first self-supervised 3D representation model learned from point maps. Point maps encode explicit 3D coordinates on a structured 2D grid, preserving global 3D geometry while remaining compatible with the input format of 2D foundation models. To transfer rich 2D priors into POMA-3D, a view-to-scene alignment strategy is designed. Moreover, as point maps are view-dependent with respect to a canonical space, we introduce POMA-JEPA, a joint embedding-predictive architecture that enforces geometrically consistent point map features across multiple views. Additionally, we introduce ScenePoint, a point map dataset constructed from 6.5K room-level RGB-D scenes and 1M 2D image scenes to facilitate large-scale POMA-3D pretraining. Experiments show that POMA-3D serves as a strong backbone for both specialist and generalist 3D understanding. It benefits diverse tasks, including 3D question answering, embodied navigation, scene retrieval, and embodied localization, all achieved using only geometric inputs (i.e., 3D coordinates). Overall, our POMA-3D explores a point map way to 3D scene understanding, addressing the scarcity of pretrained priors and limited data in 3D representation learning. Project Page: https://matchlab-imperial.github.io/poma3d/

Figures (5)

• Figure 1: Overview of POMA-3D. POMA-3D is a self-supervised 3D model pretrained on the large-scale point map dataset ScenePoint via alignment with 2D foundation models and the POMA-JEPA objective. The 3D features from pretrained POMA-3D transfer effectively to diverse 3D understanding tasks, including 3D visual question answering, embodied navigation, scene retrieval, and embodied localization.
• Figure 2: Overview of the POMA-3D pretraining. POMA-3D is pretrained with two objectives: (1) aligning [CLS] embeddings from the point map context encoder with image and text embeddings from the frozen FG-CLIP using $\mathcal{L}_{\text{view}}$ and $\mathcal{L}_{\text{scene}}$, and (2) reconstructing masked point map embeddings from the target encoder using unmasked embeddings from the context encoder via a predictor optimized by $\mathcal{L}_{\text{pjepa}}$. The target encoder is updated via EMA of the context encoder, and its final weights are used for downstream 3D understanding.
• Figure 3: Qualitative scene retrieval results. Top-4 candidates from each method are shown. For the given query, only POMA-3D retrieves the unique ground-truth scene, while others fail to return bookshelf-containing scenes. Green boxes mark bookshelves.
• Figure 4: Qualitative embodied localization results. Top: text to describe the current agent's situation. Bottom: merged multi-view point maps, where red regions indicate the point map views retrieved by POMA-3D based on the text.
• Figure 5: Comparison of POMA-3D and its baseline FG-CLIP with varying numbers of views on SQA3D ma2022sqa3d.