Point Cloud as a Foreign Language for Multi-modal Large Language Model

TL;DR

This work presents SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder, and introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens.

Abstract

Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens--treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Furthermore, to enhance the model's reasoning capability on complex 3D tasks, we propose a preference optimization training strategy with a semantic alignment-based reward, specifically designed for open-ended 3D question answering where responses are descriptive. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency, generalization across LLM backbones, and robustness to input resolution variations. Code is available at: github.com/snehaputul/SAGE3D.

Figures (6)

• Figure 1: Our proposed encoder-free 3D Multimodal Large Language Model efficiently captures 3D information from point clouds without relying on any pretrained 3D encoder. The figure on the left illustrates the overall architecture, while the figure on the right shows an example conversation about an object generated by our model.
• Figure 2: Architecture of our proposed method, encoder-free 3D Multimodal Large Language Model.
• Figure 3: The proposed training pipeline of our model. The model is trained in three stages --- each stage focusing on a specific training objective.
• Figure 4: Performance of SAGE on diverse ranges of point cloud resolution on 3D captioning task on Objaverse dataset.
• Figure A1: Fig. (Left) Impact of the number of LLM trainable layers during the stage 1 training. Fig. (Middle) Impact of the number of LLM trainable layers during stage 1 training. Fig(Right) Impact of group normalization coefficient on performance of SAGE-7B.