On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation

TL;DR

This work targets the high cost of adapting large pre-trained 3D point-cloud transformers to downstream tasks. It introduces Geometry Encoding Mixer (GEM), a geometry-aware PEFT module that combines a Spatial Adapter for local geometry refinement and a Context Adapter that uses a small set of latent tokens to inject global scene context. GEM achieves performance comparable to or better than full fine-tuning while updating only about $1.6\%$ of parameters, and demonstrates strong data efficiency, decoder compatibility, and applicability across indoor and outdoor datasets. These results underscore the value of explicitly modeling geometric cues and global context for scalable, efficient adaptation of large-scale 3D vision models.

Abstract

The emergence of large-scale pre-trained point cloud models has significantly advanced 3D scene understanding, but adapting these models to specific downstream tasks typically demands full fine-tuning, incurring high computational and storage costs. Parameter-efficient fine-tuning (PEFT) techniques, successful in natural language processing and 2D vision tasks, would underperform when naively applied to 3D point cloud models due to significant geometric and spatial distribution shifts. Existing PEFT methods commonly treat points as orderless tokens, neglecting important local spatial structures and global geometric contexts in 3D modeling. To bridge this gap, we introduce the Geometric Encoding Mixer (GEM), a novel geometry-aware PEFT module specifically designed for 3D point cloud transformers. GEM explicitly integrates fine-grained local positional encodings with a lightweight latent attention mechanism to capture comprehensive global context, thereby effectively addressing the spatial and geometric distribution mismatch. Extensive experiments demonstrate that GEM achieves performance comparable to or sometimes even exceeding full fine-tuning, while only updating 1.6% of the model's parameters, fewer than other PEFT methods. With significantly reduced training time and memory requirements, our approach thus sets a new benchmark for efficient, scalable, and geometry-aware fine-tuning of large-scale 3D point cloud models. Code is available at https://github.com/LiyaoTang/GEM.

Figures (4)

• Figure 1: (a) Existing PEFT methods, such as adapters, prompt tuning, and LoRA, focus on adaptations inside attention and feed-forward layers. (b) In contrast, Geometry Encoding Mixer (GEM) explicitly encodes the geometric cues and mixes them into the pre-trained model, by the Spatial Adapter refining the pre-trained positional encoding and the Context Adapter complementing the local attention. (c) By capturing 3D spatial details and scene-wide geometry context, GEM reaches full fine-tuning performance while tuning $<2\%$ parameters, outperforming existing PEFT methods.
• Figure 2: Geometry Encoding Mixed. We propose the spatial adapter to enhance the pre-trained positional encoding, and the context adapter to overcome the local attention mechanism, thus enhancing the efficient adaptation on large-scale 3D scenes with explicit geometry encoding.
• Figure 3: We visualize the attention weights of our latent tokens, comparing to the attentional weights with the prompt tuning, showing the enhanced geometry cues produced by GEM.
• Figure 4: Attention maps of our latent tokens in context adapter.