CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding

TL;DR

CoT3DRef addresses the challenge of interpretable, data-efficient 3D visual grounding by formulating grounding as a Seq2Seq task that first predicts a chain of anchors $O^T$ and then the target, guided by a Pathway module that yields a logical order $O^P$. A Transformer-based Chain-of-Thoughts decoder enforces stepwise, causal reasoning over multi-modal features, and a fully automatic pseudo-label generator provides inexpensive supervision without manual annotation. The approach yields state-of-the-art results on Nr3D, Sr3D, and ScanRefer while maintaining data efficiency, including matching Sr3D's SOTA with only 10% of the data. The method is modular and readily integrable with existing architectures, offering interpretability and potential impact for robotics, assistive tech, and federated learning where labeled data are scarce.

Abstract

3D visual grounding is the ability to localize objects in 3D scenes conditioned by utterances. Most existing methods devote the referring head to localize the referred object directly, causing failure in complex scenarios. In addition, it does not illustrate how and why the network reaches the final decision. In this paper, we address this question Can we design an interpretable 3D visual grounding framework that has the potential to mimic the human perception system?. To this end, we formulate the 3D visual grounding problem as a sequence-to-sequence Seq2Seq task by first predicting a chain of anchors and then the final target. Interpretability not only improves the overall performance but also helps us identify failure cases. Following the chain of thoughts approach enables us to decompose the referring task into interpretable intermediate steps, boosting the performance and making our framework extremely data-efficient. Moreover, our proposed framework can be easily integrated into any existing architecture. We validate our approach through comprehensive experiments on the Nr3D, Sr3D, and Scanrefer benchmarks and show consistent performance gains compared to existing methods without requiring manually annotated data. Furthermore, our proposed framework, dubbed CoT3DRef, is significantly data-efficient, whereas on the Sr3D dataset, when trained only on 10% of the data, we match the SOTA performance that trained on the entire data. The code is available at https:eslambakr.github.io/cot3dref.github.io/.

Figures (13)

• Figure 1: Overview of our approach, where we first predict a chain of anchors in a logical order. In this example, to reach the chair target, we first have to localize the white and red boxes, then the bookshelf.
• Figure 2: Data efficiency results. To show the effectiveness of our CoT architecture, we integrate it into four architectures, i.e., MVT huang2022multi, SAT yang2021sat, LAR bakr2022look and ViL chen2022language, across different amounts of training data (10% - 100%).
• Figure 3: An overview of our Chain-of-Thoughts Data-Efficient 3D visual grounding framework (CoT3DRef). First, we predict the anchors $\mathcal{O^T}$ from the input utterance, then sort the anchors in a logical order $\mathcal{O^P}$ using the Pathway module. Then, we feed the multi-modal features $\mathcal{F}$, the parallel localized objects $\mathcal{R^F}$, and the logical path $\mathcal{O^P}$ to our Chain-of-Thoughts decoder to localize the referred object and the anchors in a logical order $\mathcal{R^P}$.
• Figure 4: Our qualitative results. First row indicates the GT w.r.t the input utterance, demonstrated in the last row. Second and third rows show the qualitative results for MVT and our method, respectively. The success and the failure cases are shown in green and red boxes, respectively.
• Figure 5: Identification of failure cases as a benefit of the interpretability. In this example, there are two anchors mentioned in the description, desk and monitor, and the target is the chair. The correct chair should be number two, however, the model predicts number four. By visualizing the attention maps, on the left, we can identify the main cause of the wrong prediction, whereas, the first anchor localize a wrong desk (desk #3). Therefore, the rest of the chain, i.e., the monitor and the chair are localized wrongly.