Exploring the Potential of Encoder-free Architectures in 3D LMMs

TL;DR

This work investigates encoder-free architectures for 3D large multimodal models by removing traditional 3D encoders and transferring their role to the language model. It introduces two key strategies—LLM-embedded Semantic Encoding and Hierarchical Geometry Aggregation—to embed high-level 3D semantics and capture local geometric details within the LLM framework, resulting in the ENEL model that achieves competitive results on 3D captioning, classification, and VQA while offering efficiency gains. The findings demonstrate the viability of encoder-free designs for 3D understanding and provide practical guidance through extensive ablations and visualizations. Overall, ENEL indicates a promising direction for lightweight, end-to-end 3D LMMs that align closely with LLM capabilities.

Abstract

Encoder-free architectures have been preliminarily explored in the 2D Large Multimodal Models (LMMs), yet it remains an open question whether they can be effectively applied to 3D understanding scenarios. In this paper, we present the first comprehensive investigation into the potential of encoder-free architectures to alleviate the challenges of encoder-based 3D LMMs. These long-standing challenges include the failure to adapt to varying point cloud resolutions during inference and the point features from the encoder not meeting the semantic needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to remove the pre-trained encoder and enable the LLM to assume the role of the 3D encoder: 1) We propose the LLM-embedded Semantic Encoding strategy in the pre-training stage, exploring the effects of various point cloud self-supervised losses. And we present the Hybrid Semantic Loss to extract high-level semantics. 2) We introduce the Hierarchical Geometry Aggregation strategy in the instruction tuning stage. This incorporates inductive bias into the LLM layers to focus on the local details of the point clouds. To the end, we present the first Encoder-free 3D LMM, ENEL. Our 7B model rivals the state-of-the-art model, PointLLM-PiSA-13B, achieving 57.91%, 61.0%, and 55.20% on the classification, captioning, and VQA tasks, respectively. Our results show that the encoder-free architecture is highly promising for replacing encoder-based architectures in the field of 3D understanding. The code is released at https://github.com/Ivan-Tang-3D/ENEL

Figures (11)

• Figure 1: Issues of encoder-based 3D LMMs. (a) Point Cloud Resolution Limitation. During training, the point cloud size (P.C. Size) and point token size (P.T. Size) are fixed at 8192 and 512, respectively. And we adjust these two sizes during inference, point cloud size from 2K to 16K and the corresponding point token size from 128 to 2048. We evaluate them on the captioning task of the Objaverse benchmark using GPT-4 score as the evaluation metric. (b) Embedding Semantic Discrepancy. We visualize the attention scores of the average text token to the point tokens, where red indicates higher values. The point tokens in the encoder-free architecture exhibit stronger textual semantic relevance needed for the LLM.
• Figure 2: Token Embedding. Performance on Objaverse with PointLLM-7B as the baseline. 'Cls'/'Cap': classification/captioning tasks. 'Avg': accuracy under prompts "What is this?" and "This is an object of." 'S-BERT': Sentence-BERT. 'T.E.': our designed token embedding module.
• Figure 3: Learnable Layers. We set the LLM early layers to be learnable. 'LR' represents the learning rate during the pre-training stage, with the original learning rate set to 2e-3.
• Figure 4: Overall Pipeline of Enel. The training is divided into two stages: the pre-training stage and the instruction tuning stage. In the first stage, we set the first $K$ layers to be learnable and apply the proposed Hybrid Semantic Loss to embed high-level semantics into the LLM. In the second stage, we adopt the Hierarchical Geometric Aggregation strategy to capture local structures of point clouds.
• Figure 5: Point Cloud Self-Supervised Learning Losses. In the pre-training stage, we explore common self-supervised learning losses for the encoder-free 3D LMM: (a) Masked Modeling Loss, (b) Reconstruction Loss, (c) Contrastive Loss, and (d) Knowledge Distillation Loss. The (e) represents our proposed Hybrid Semantic Loss, specifically designed for the encoder-free architecture.