Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence

Hao Tang, Ting Huang, Zeyu Zhang

TL;DR

3D CoCa v2 tackles the challenge of generalizable 3D captioning by unifying contrastive vision–language learning with 3D caption generation and introducing an inference-only Test-Time Search (TTS) that leverages a compact scene summary and an external LLM judge. The architecture uses a frozen CLIP-based semantic prior, a geometry-aware 3D scene encoder, and a multimodal decoder trained with both contrastive and captioning losses, avoiding external detectors. Empirical results show consistent in-domain gains on ScanRefer and Nr3D and improved zero-shot robustness on TOD$^3$Cap, with TTS providing up to +3.6 CIDEr in OOD settings and a controllable speed-accuracy trade-off. The work demonstrates that plug-in inference-time search can significantly enhance faithfulness and grounding under distribution shifts, offering a practical path toward robust 3D vision–language systems for embodied applications.

Abstract

Spatial intelligence refers to the ability to perceive, reason about, and describe objects and their relationships within three-dimensional environments, forming a foundation for embodied perception and scene understanding. 3D captioning aims to describe 3D scenes in natural language; however, it remains challenging due to the sparsity and irregularity of point clouds and, more critically, the weak grounding and limited out-of-distribution (OOD) generalization of existing captioners across drastically different environments, including indoor and outdoor 3D scenes. To address this challenge, we propose 3D CoCa v2, a generalizable 3D captioning framework that unifies contrastive vision-language learning with 3D caption generation and further improves robustness via test-time search (TTS) without updating the captioner parameters. 3D CoCa v2 builds on a frozen CLIP-based semantic prior, a spatially-aware 3D scene encoder for geometry, and a multimodal decoder jointly optimized with contrastive and captioning objectives, avoiding external detectors or handcrafted proposals. At inference, TTS produces diverse caption candidates and performs reward-guided selection using a compact scene summary. Experiments show improvements over 3D CoCa of +1.50 CIDEr@0.5IoU on ScanRefer and +1.61 CIDEr@0.5IoU on Nr3D, and +3.8 CIDEr@0.25 in zero-shot OOD evaluation on TOD3Cap. Code will be released at https://github.com/AIGeeksGroup/3DCoCav2.

3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence

TL;DR

3D CoCa v2 tackles the challenge of generalizable 3D captioning by unifying contrastive vision–language learning with 3D caption generation and introducing an inference-only Test-Time Search (TTS) that leverages a compact scene summary and an external LLM judge. The architecture uses a frozen CLIP-based semantic prior, a geometry-aware 3D scene encoder, and a multimodal decoder trained with both contrastive and captioning losses, avoiding external detectors. Empirical results show consistent in-domain gains on ScanRefer and Nr3D and improved zero-shot robustness on TODCap, with TTS providing up to +3.6 CIDEr in OOD settings and a controllable speed-accuracy trade-off. The work demonstrates that plug-in inference-time search can significantly enhance faithfulness and grounding under distribution shifts, offering a practical path toward robust 3D vision–language systems for embodied applications.

Abstract

Spatial intelligence refers to the ability to perceive, reason about, and describe objects and their relationships within three-dimensional environments, forming a foundation for embodied perception and scene understanding. 3D captioning aims to describe 3D scenes in natural language; however, it remains challenging due to the sparsity and irregularity of point clouds and, more critically, the weak grounding and limited out-of-distribution (OOD) generalization of existing captioners across drastically different environments, including indoor and outdoor 3D scenes. To address this challenge, we propose 3D CoCa v2, a generalizable 3D captioning framework that unifies contrastive vision-language learning with 3D caption generation and further improves robustness via test-time search (TTS) without updating the captioner parameters. 3D CoCa v2 builds on a frozen CLIP-based semantic prior, a spatially-aware 3D scene encoder for geometry, and a multimodal decoder jointly optimized with contrastive and captioning objectives, avoiding external detectors or handcrafted proposals. At inference, TTS produces diverse caption candidates and performs reward-guided selection using a compact scene summary. Experiments show improvements over 3D CoCa of +1.50 CIDEr@0.5IoU on ScanRefer and +1.61 CIDEr@0.5IoU on Nr3D, and +3.8 CIDEr@0.25 in zero-shot OOD evaluation on TOD3Cap. Code will be released at https://github.com/AIGeeksGroup/3DCoCav2.
Paper Structure (20 sections, 15 equations, 5 figures, 8 tables, 2 algorithms)

This paper contains 20 sections, 15 equations, 5 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Method
  4. Overview
  5. 3D Scene Encoder
  6. Text Encoder
  7. Contrastive Learning
  8. Multi-Modal Fusion Decoder
  9. Training Objectives and Joint Optimization
  10. Test-Time Search
  11. Experiments
  12. Datasets and Evaluation Metrics
  13. Implementation Details
  14. In-Domain Evaluation
  15. Out-of-Domain Evaluation
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overview of 3D CoCa v2 and OOD results on TOD$^3$Cap. (a) 3D CoCa v2 extends 3D CoCa huang20253dcoca with an inference-only test-time search (TTS) module and an external LLM judge. (b) Zero-shot OOD performance on TOD$^3$Cap todocap2025 comparing 3D-VLP 3dvlp2024, 3D CoCa huang20253dcoca, and 3D CoCa v2 under standard captioning metrics at IoU 0.25 and 0.5.
  • Figure 2: Overview of 3D CoCa v2. (a) 3D CoCa learns aligned 3D–text representations by jointly optimizing contrastive alignment and caption generation: a point-cloud scene encoder and a text encoder produce fused features for a multi-modal decoder to generate a draft caption. (b) Test-Time Search (inference-only) improves robustness without any parameter updates by generating best-of-$N$ candidate captions from the backbone, conditioning an external LLM judge on a compact scene summary, and selecting the highest-scoring candidate as the final caption.
  • Figure 3: Qualitative comparisons on ScanRefer chen2020scanrefer. We visualize four representative scenes and the captions generated by 3D CoCa, 3D CoCa v2, and the ground truth (GT). Compared to the baseline, 3D CoCa v2 produces more detailed and better-grounded descriptions, capturing richer scene semantics and functional cues. Red-highlighted phrases mark the additional informative content provided by our method beyond the baseline.
  • Figure 4: Qualitative results on TOD$^3$Cap todocap2025 (OOD, zero-shot). We compare captions generated by the indoor-trained Vote2Cap-DETR++, 3D CoCa and 3D CoCa v2 on outdoor scenes with paired front and back views. Vote2Cap-DETR++ and 3D CoCa often exhibit a strong indoor bias, producing generic indoor descriptions, whereas 3D CoCa v2 generates more scene-consistent outdoor captions that better reflect key semantics. Ground-truth (GT) captions are shown for reference. Red words highlight informative details captured by 3D CoCa v2 but missing in the baseline.
  • Figure 5: Qualitative comparisons on ScanRefer chen2020scanrefer (w/o TTS vs w/ TTS). For each example, we show the reconstructed 3D scene (top) and a zoomed-in view (bottom), where the target object is indicated by the magenta box. Compared with standard decoding (w/o TTS), Test-Time Search (w/ TTS) yields more specific and better-grounded captions, capturing object identities and layout cues supported by the highlighted region rather than generic room-level descriptions. Green text marks the object-specific details introduced by w/ TTS.