CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation

TL;DR

CMMCoT introduces a memory-augmented, slow-thinking framework for complex multi-image understanding, addressing limitations of end-to-end multimodal prediction. It combines interleaved multimodal sequence representations with a Retrieval-based Image Feature Reasoning Enhancement Module (RIFREM) to enable dynamic cross-image reasoning during inference. A new CMMCoT-260K dataset provides four reasoning task types (Caption, Co-reference, Comparison, Reason) to train and evaluate multi-image CoT. Across six benchmarks, including multi-image and single-image tasks, CMMCoT achieves state-of-the-art results and offers improved interpretability of intermediate reasoning steps. The work highlights the value of structured, memory-augmented reasoning for robust and explainable multi-modal AI systems.

Abstract

While previous multimodal slow-thinking methods have demonstrated remarkable success in single-image understanding scenarios, their effectiveness becomes fundamentally constrained when extended to more complex multi-image comprehension tasks. This limitation stems from their predominant reliance on text-based intermediate reasoning processes. While for human, when engaging in sophisticated multi-image analysis, they typically perform two complementary cognitive operations: (1) continuous cross-image visual comparison through region-of-interest matching, and (2) dynamic memorization of critical visual concepts throughout the reasoning chain. Motivated by these observations, we propose the Complex Multi-Modal Chain-of-Thought (CMMCoT) framework, a multi-step reasoning framework that mimics human-like "slow thinking" for multi-image understanding. Our approach incorporates two key innovations: (1) The construction of interleaved multimodal multi-step reasoning chains, which utilize critical visual region tokens, extracted from intermediate reasoning steps, as supervisory signals. This mechanism not only facilitates comprehensive cross-modal understanding but also enhances model interpretability. (2) The introduction of a test-time memory augmentation module that expands the model's reasoning capacity during inference while preserving parameter efficiency. Furthermore, to facilitate research in this direction, we have curated a novel multi-image slow-thinking dataset. Extensive experiments demonstrate the effectiveness of our model. Code is available at https://github.com/zhangguanghao523/CMMCoT.

Figures (7)

• Figure 1: Data generation process of our proposed CMMCoT dataset. The construction of the CMMCoT dataset mainly concerns four parts: the generation of QA rationale chains, the extraction of textual entities, the detection and validation of visual entities, and the spatial fusion and summarization of entity groundings.
• Figure 2: Illustration of the overall framework of CMMCoT. Part (a) depicts the training structure, with the input and output shown below and above the model, respectively. Part (b) presents the inference structure of the model, where the memory bank stores the K and V for each layer of the input images. The RIFREM module is integrated between different decoder layers during inference. Part (c) represents the detailed structure of the RIFREM module.
• Figure 3: Visualization results of our CMMCoT task to illustrate the difference between our model and previous methods.
• Figure 4: Ablation study on the number of RIFREM module layers: The red line represents the performance effects, while the blue line indicates the impact on latency.
• Figure 5: Visualization results of our CMMCoT task to illustrate the fine-grained reasoning ability of our model.