AIMCoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

TL;DR

This paper tackles the unreliability of attention-based region selection in vision-language CoT by reframing visual evidence gathering as active information-foraging. It introduces AIMCoT, a framework composed of Context-enhanced Attention-map Generation (CAG), Active Visual Probing (AVP), and Dynamic Attention-shifting Trigger (DAT) to proactively discover and insert informative visual cues at cognitively critical moments. Empirical results across M3CoT, ScienceQA, and LLaVA-W on Chameleon-7B and Qwen2-VL-7B-Instruct backbones show substantial improvements over state-of-the-art baselines in both 0- and 1-shot settings, with particularly large gains on open-ended benchmarks. By enabling a human-like, goal-directed reasoning process, AIMCoT advances robust multimodal reasoning and provides reproducible code and deep analyses of component interactions and deployment considerations.

Abstract

Multimodal Chain-of-Thought (CoT) has emerged as a powerful technique for enhancing the vision-language reasoning with interleaved information. However, existing methods often rely on simplistic heuristics for constructing interleaved CoT, typically depending on attention maps, which our empirical analysis reveals can be unreliable. What's more, the shortcomings of their passive and purposeless selection strategies and their arbitrary triggering mechanisms in capturing the model's cognitive need for information are further amplified. In this paper, we propose \textbf{AIMCoT}, an \textbf{A}ctive \textbf{I}nformation-driven \textbf{M}ulti-modal \textbf{C}hain-\textbf{o}f-\textbf{T}hought framework that addresses these fundamental limitations. AIMCoT introduces three synergistic components: (1) \textbf{Context-enhanced Attention-map Generation (CAG)}, which mitigates the text-vision granularity imbalance, thereby producing more reliable attention maps as a foundation. (2) \textbf{Active Visual Probing (AVP)}, which replaces passive selection with a proactive, goal-oriented strategy grounded in information theory to select image regions that help answer the questions maximally. (3) \textbf{Dynamic Attention-shifting Trigger (DAT)}, which intelligently determines the optimal moments to insert visual information by monitoring the model's text-to-vision attention shifts. Extensive experiments on three challenging benchmarks demonstrate that AIMCoT significantly outperforms state-of-the-art methods across different settings. By actively foraging for information and dynamically structuring its reasoning process, AIMCoT represents a critical step towards more robust, effective, and human-like multimodal reasoning. Our code is available at https://anonymous.4open.science/r/AIMCoT.

Figures (5)

• Figure 1: The images of the 22nd question on LLaVA-W benchmark, which is a close-up photo of a meal at ICHIRAN. The left and right figures are respectively the original image and the first three sets of regions selected by the Top-K strategy (red, purple, and blue, respectively). A detailed explanation is shown in Appendix \ref{['app: explanation for fig 1']}.
• Figure 2: The visualization of regions selected by the information gain-guided strategy.
• Figure 3: An overview of our AVP module, which iteratively selects $K$ most informative regions from a diversified candidate set $C$ to build an interleaved CoT that fosters vision-language reasoning.
• Figure 4: An illustration of the entire process of context enhancement by the CAG module, using problem physical-commonsense-1398 from the M3CoT benchmark as an example. This showcases both the template and usage of $\mathcal{P}_{CAG}$.
• Figure 5: Experimental results of the sensitivity analysis of the hyper-parameter $\delta$. The left figure illustrates the performance of AIMCoT when $\delta$ takes different values, while the right one shows the number of times the AVP module within AIMCoT is triggered.