Rethinking Multimodal Few-Shot 3D Point Cloud Segmentation: From Fused Refinement to Decoupled Arbitration

TL;DR

This work addresses the Plasticity-Stability Dilemma in multimodal few-shot 3D point cloud segmentation by decoupling geometry and semantics into separate, mutually regularized pathways. The proposed DA-FSS framework introduces Parallel Experts Refinement, a Decoupled Alignment Module (DAM), and a Stacked Arbitration Module (SAM) to prevent gradient domination from frozen semantic priors while leveraging textual guidance. Key innovations include Prototype Loss Regularization (PLR) and Decoupled Consistency Regularization (DCR) to align and stabilize cross-path representations, along with boundary-guided fusion. Empirical results on S3DIS and ScanNet show consistent improvements over state-of-the-art methods with better geometric completeness and texture differentiation, while maintaining or reducing computational overhead. This approach offers a principled strategy for extracting and fusing multimodal priors in 3D scene understanding and paves the way for more robust, open-world FS-PCS systems.

Abstract

In this paper, we revisit multimodal few-shot 3D point cloud semantic segmentation (FS-PCS), identifying a conflict in "Fuse-then-Refine" paradigms: the "Plasticity-Stability Dilemma." In addition, CLIP's inter-class confusion can result in semantic blindness. To address these issues, we present the Decoupled-experts Arbitration Few-Shot SegNet (DA-FSS), a model that effectively distinguishes between semantic and geometric paths and mutually regularizes their gradients to achieve better generalization. DA-FSS employs the same backbone and pre-trained text encoder as MM-FSS to generate text embeddings, which can increase free modalities' utilization rate and better leverage each modality's information space. To achieve this, we propose a Parallel Expert Refinement module to generate each modal correlation. We also propose a Stacked Arbitration Module (SAM) to perform convolutional fusion and arbitrate correlations for each modality pathway. The Parallel Experts decouple two paths: a Geometric Expert maintains plasticity, and a Semantic Expert ensures stability. They are coordinated via a Decoupled Alignment Module (DAM) that transfers knowledge without propagating confusion. Experiments on popular datasets (S3DIS, ScanNet) demonstrate the superiority of DA-FSS over MM-FSS. Meanwhile, geometric boundaries, completeness, and texture differentiation are all superior to the baseline. The code is available at: https://github.com/MoWenQAQ/DA-FSS.

Figures (4)

• Figure 1: Visual comparison of the Plasticity-Stability Dilemma. (a) Ground Truth.(b) The baseline MM-FSS an2024multimodality suffers from Gradient Domination, where strong semantic priors suppress geometric adaptation.This leads to semantic hallucinations in texture-ambiguous regions (e.g., mistaking the white wall for a "Picture" in ScanNet, top row) or geometric corruption (e.g., losing structural details of the "Table" in S3DIS, bottom row).(c) Our DA-FSS structurally decouples the experts, effectively eliminating texture-induced noise and preserving intrinsic 3D integrity.
• Figure 2: The Baseline (Orange)’s gradient norm declines rapidly with high volatility, characterized by diminishing gradients due to semantic domination. In contrast, our DA-FSS (Purple)’s gradient norm maintains a relatively stable trend.
• Figure 3: Overall architecture of the proposed DA-FSS. Given support and query point clouds, we first generate intermodal correlations $F_{s/q}^{i}$ from the IF head and unimodal correlations $F_{s/q}^{u}$ from the UF head. These correlations are then forwarded to the Parallel Experts (Geometric Expert $\mathcal{T}_{geo}$ and Semantic Expert $\mathcal{T}_{sem}$) to independently refine features, isolating plasticity from stability. Moreover, we use the Decoupled Alignment Module (DAM) to align correlations ($C_i, C_u$) via PLR and refined experts ($P_{sem}, P_{geo}$) via DCR only in training. Finally, we propose the Stacked Arbitration Module (SAM), which synthesizes the final decision by leveraging boundary-injected guidance ($G_{base}$ and $G_q$) to effectively arbitrate between geometric and semantic pathways after a 1$\times$1 convolutional fusion. For clarity, we present the model under the 1-way 1-shot setting.
• Figure 4: Qualitative comparison between MM-FSS and our proposed DA-FSS in the 1-way 1-shot setting on the ScanNet dataset split 1. The target classes in the columns are curtain, counter, sink, table, wall and desk, respectively. Comparison between (b) Baseline and (c) Ours demonstrates a fundamental trade-off. It is evident that Ours (DA-FSS) exhibits a significant advantage in the geometric completeness of classes and can effectively repair instances of missing points. Moreover, on counter classes, Baseline (MM-FSS) chooses to directly ignore them, whereas Ours (DA-FSS) is capable of recognizing them. However, Ours occasionally exhibits boundary over-segmentation to ensure geometric completeness.