Watch Wider and Think Deeper: Collaborative Cross-modal Chain-of-Thought for Complex Visual Reasoning

TL;DR

This work tackles cross-modal reasoning with chain-of-thought by addressing two core weaknesses: reliance on a single image region and fragmentation across reasoning steps. It introduces CoCoT, combining Dynamic Multi-Region Grounding and Relation-Aware Reasoning to enable collaborative region use and coherent thought chains, validated via a newly constructed CoCoT-70K dataset of $74{,}691$ samples across six benchmarks. The approach employs a two-stage training paradigm and achieves substantial gains, including $15.4\%$ accuracy improvement on LLaVA-1.5 and $4.0\%$ on Qwen2-VL, with ablations illustrating the necessity of separating grounding from relational reasoning. The dataset and method advance multimodal chain-of-thought research and offer a practical foundation for improving complex visual reasoning in vision-language models.

Abstract

Multi-modal reasoning requires the seamless integration of visual and linguistic cues, yet existing Chain-of-Thought methods suffer from two critical limitations in cross-modal scenarios: (1) over-reliance on single coarse-grained image regions, and (2) semantic fragmentation between successive reasoning steps. To address these issues, we propose the CoCoT (Collaborative Coross-modal Thought) framework, built upon two key innovations: a) Dynamic Multi-Region Grounding to adaptively detect the most relevant image regions based on the question, and b) Relation-Aware Reasoning to enable multi-region collaboration by iteratively aligning visual cues to form a coherent and logical chain of thought. Through this approach, we construct the CoCoT-70K dataset, comprising 74,691 high-quality samples with multi-region annotations and structured reasoning chains. Extensive experiments demonstrate that CoCoT significantly enhances complex visual reasoning, achieving an average accuracy improvement of 15.4% on LLaVA-1.5 and 4.0% on Qwen2-VL across six challenging benchmarks. The data and code are available at: https://github.com/deer-echo/CoCoT.

Figures (4)

• Figure 1: Single-region CoT vs. CoCoT.
• Figure 2: Examples of six datasets in the CoCoT-70K dataset.
• Figure 3: Overview of CoCoT.
• Figure 4: Data statistic of CoCoT-70k.