Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learning

TL;DR

This work tackles the challenge of robust 3D vision-language alignment by integrating structured Chain-of-Thought (CoT) reasoning into training. It introduces the 3D-CoT Benchmark, a large-scale dataset with hierarchical CoT annotations across shape recognition, functional inference, and causal reasoning, augmented from existing 3D corpora. A dual-layer evaluation framework separately measures intermediate reasoning quality and final inference accuracy, and a two-stage learning protocol aligns 3D representations with text before integrating reasoning signals. Findings show CoT substantially improves 3D semantic grounding, with large reasoning models exploiting CoT more effectively than general LLMs; annotation style further modulates performance, with unmarked CoT favoring LRMs and tagged CoT benefiting LLMs. Overall, CoT emerges as a fundamental mechanism to bridge geometry and semantics in multimodal 3D tasks, with broad implications for future cross-modal reasoning systems and embodied AI.

Abstract

Chain-of-Thought (CoT) reasoning has proven effective in natural language tasks but remains underexplored in multimodal alignment. This study investigates its integration into 3D vision-language learning by embedding structured reasoning into alignment training. We introduce the 3D-CoT Benchmark, a dataset with hierarchical CoT annotations covering shape recognition, functional inference, and causal reasoning. Through controlled experiments, we compare CoT-structured and standard textual annotations across large reasoning models (LRMs) and large language models (LLMs). Our evaluation employs a dual-layer framework assessing both intermediate reasoning and final inference quality. Extensive experiments demonstrate that CoT significantly improves 3D semantic grounding, with LRMs leveraging CoT more effectively than LLMs. Furthermore, we highlight that annotation structure influences performance-explicit reasoning markers aid LLMs, while unmarked CoT better aligns with LRM inference patterns. Our analyses suggest that CoT is crucial for enhancing multimodal reasoning, with implications beyond 3D tasks. The dataset will be publicly available at https://huggingface.co/datasets/Battam/3D-CoT

Figures (4)

• Figure 1: Illustration of 3D vision-language reasoning. (a) Traditional 3D vision-language alignment relies on static descriptions, overlooking hierarchical reasoning. (b) Our CoT-based approach encodes intermediate reasoning steps, enhancing semantic grounding and functional inference.
• Figure 2: Overview of our Chain-of-Thought (CoT) reasoning for 3D vision-language alignment. Left: Model behavior categorized by annotation type (Tagged vs. Unmarked) and model type (LLM vs. LRM). Right: Illustration of CoT's role in enhancing reasoning depth: Tagged CoT explicitly delineates steps, while Unmarked CoT fosters implicit integration. Overall, LLMs benefit from explicit segmentation, whereas LRMs align better with unmarked reasoning.
• Figure 3: Example outputs from models evaluated on the CoT-GApartNet test set. The structured annotations facilitate part-level functional reasoning, demonstrating the dataset's effectiveness in enhancing multimodal understanding. a,c from LRM, b,d from LLM.
• Figure 4: Case study of model outputs with different annotation strategies. No CoT results in minimal descriptions, while unmarked CoT and tagged CoT produce structured multi-step reasoning. Here, the tagged CoT explicitly segments reasoning with <think> markers.