Mono3DVG-EnSD: Enhanced Spatial-aware and Dimension-decoupled Text Encoding for Monocular 3D Visual Grounding

TL;DR

The paper tackles monocular 3D grounding from text by addressing two core issues: over-reliance on explicit keywords and cross-dimensional interference when fusing generalized textual features with 2D/3D visuals. It introduces Mono3DVG-EnSD, consisting of a CLIP-Guided Lexical Certainty Adapter (CLIP-LCA) that masks high-certainty words to emphasize spatial language, and a Dimension-Decoupled Module (D2M) that decouples 2D and 3D textual features for dimensionally consistent cross-modal refinement. Empirical results on the Mono3DRefer dataset show state-of-the-art performance across nine evaluation scenarios, with a notable Far(Acc@0.5) improvement of +13.54%, and ablations confirm that both CLIP-LCA and D2M independently contribute to performance gains. The work advances robust, language-guided monocular 3D grounding, with practical implications for vision-language understanding in robotics and human-robot interaction.

Abstract

Monocular 3D Visual Grounding (Mono3DVG) is an emerging task that locates 3D objects in RGB images using text descriptions with geometric cues. However, existing methods face two key limitations. Firstly, they often over-rely on high-certainty keywords that explicitly identify the target object while neglecting critical spatial descriptions. Secondly, generalized textual features contain both 2D and 3D descriptive information, thereby capturing an additional dimension of details compared to singular 2D or 3D visual features. This characteristic leads to cross-dimensional interference when refining visual features under text guidance. To overcome these challenges, we propose Mono3DVG-EnSD, a novel framework that integrates two key components: the CLIP-Guided Lexical Certainty Adapter (CLIP-LCA) and the Dimension-Decoupled Module (D2M). The CLIP-LCA dynamically masks high-certainty keywords while retaining low-certainty implicit spatial descriptions, thereby forcing the model to develop a deeper understanding of spatial relationships in captions for object localization. Meanwhile, the D2M decouples dimension-specific (2D/3D) textual features from generalized textual features to guide corresponding visual features at same dimension, which mitigates cross-dimensional interference by ensuring dimensionally-consistent cross-modal interactions. Through comprehensive comparisons and ablation studies on the Mono3DRefer dataset, our method achieves state-of-the-art (SOTA) performance across all metrics. Notably, it improves the challenging Far(Acc@0.5) scenario by a significant +13.54%.

Figures (4)

• Figure 1: (a) Existing methods over-rely on high-certainty words (e.g., "red car") within descriptions (left), causing spatial description neglect. Our CLIP-LCA dynamically masks such high-certainty words during training, forcing the text encoder to comprehend spatial descriptors; (b) Previous methods employ generalized textual features (containing both 2D and 3D information) to refine both 3D visual (depth) features and 2D visual features, causing cross-dimensional interference. We propose D2M to decouple dimension-specific text features (2D and 3D separately) for dimensionally-consistent cross-modal interactions.
• Figure 2: Framework Overview. The architecture integrates multiple feature extraction modules: RoBERTa for generalized textual features ($T_{t}$), visual encoder for 2D visual features ($V_{2D}$), and depth encoder for 3D visual features ($V_{3D}$). CLIP-LCA dynamically adjusts the certainty level of textual descriptions during training. D2M decomposes generalized textual features ($T_{t}$) into 2D-specific ($T_{2D}$) and 3D-specific textual features ($T_{3D}$). The adapter refines dimension-specific features of target objects, followed by decoder and multi-MLPs head for 2D-3D attribute prediction.
• Figure 3: The detailed architecture of Dimension-Decoupled Module. The left part illustrates the procedure for decoupling 2D-specific textual features $T_{2D}$, while the right section presents the corresponding workflow for 3D-specific textual features $T_{3D}$.
• Figure 4: Visualization of 3D bounding box predictions from Mono3DVG-TR and our Mono3DVG-EnSD. Ground truth (red), our predictions (green), and Mono3DVG-TR predictions (blue) are shown for comparison.