Foundational Models for 3D Point Clouds: A Survey and Outlook

TL;DR

This survey analyzes how vision-language models (VLMs) and large language models (LLMs) can underpin foundation models for 3D point clouds, organizing methods into direct adaptation, dual encoders, and triplet alignment. It covers datasets, pre-training, and downstream adaptation, and reviews 2DFM-driven approaches for 3D classification, segmentation, and detection, as well as scene-level LVLMs that integrate language with 3D reasoning. Key contributions include a cohesive taxonomy, synthesis of object- and scene-level 3D LVLMs, and critical outlooks on data, scalability, and continual learning for robust 3D understanding. The work highlights practical pathways for leveraging abundant 2D data and language priors to overcome 3D data scarcity, enabling more capable, open-world 3D AI systems with real-world impact across robotics, AR/VR, and autonomous sensing.

Abstract

The 3D point cloud representation plays a crucial role in preserving the geometric fidelity of the physical world, enabling more accurate complex 3D environments. While humans naturally comprehend the intricate relationships between objects and variations through a multisensory system, artificial intelligence (AI) systems have yet to fully replicate this capacity. To bridge this gap, it becomes essential to incorporate multiple modalities. Models that can seamlessly integrate and reason across these modalities are known as foundation models (FMs). The development of FMs for 2D modalities, such as images and text, has seen significant progress, driven by the abundant availability of large-scale datasets. However, the 3D domain has lagged due to the scarcity of labelled data and high computational overheads. In response, recent research has begun to explore the potential of applying FMs to 3D tasks, overcoming these challenges by leveraging existing 2D knowledge. Additionally, language, with its capacity for abstract reasoning and description of the environment, offers a promising avenue for enhancing 3D understanding through large pre-trained language models (LLMs). Despite the rapid development and adoption of FMs for 3D vision tasks in recent years, there remains a gap in comprehensive and in-depth literature reviews. This article aims to address this gap by presenting a comprehensive overview of the state-of-the-art methods that utilize FMs for 3D visual understanding. We start by reviewing various strategies employed in the building of various 3D FMs. Then we categorize and summarize use of different FMs for tasks such as perception tasks. Finally, the article offers insights into future directions for research and development in this field. To help reader, we have curated list of relevant papers on the topic: https://github.com/vgthengane/Awesome-FMs-in-3D.

Figures (5)

• Figure 1: Taxonomy of foundational models for 3D point clouds.
• Figure 2: A general pipeline of using FMs for 3D vision tasks.
• Figure 3: Methods of building 3D models for representation learning by leveraging 2DFMs, categorised by how FMs are used.
• Figure 4: High level overview of open vocabulary methods for point clouds using 2DFMs.
• Figure 5: High level overview of scene level methods for 3D understanding, categorised by how point cloud data is handled.