Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models

Wenxin Zhu, Andong Chen, Yuchen Song, Kehai Chen, Conghui Zhu, Ziyan Chen, Tiejun Zhao

TL;DR

The paper surveys Multimodal Chain-of-Thought (MCoT) in Multimodal Large Language Models, detailing why explicit cross-modal reasoning improves performance and interpretability. It categorizes methods into CoT paradigms, post-training data/objectives, and inference-time techniques, and analyzes the underlying mechanisms of information representation, structured reasoning, and process supervision. It also catalogs evaluation benchmarks across math, spatiotemporal, vision-language transformation, sequential/multi-image, and integrated reasoning, and surveys applications in embodied AI, autonomous driving, healthcare, multimodal generation, translation, and social affective tasks. Finally, it discusses robustness, security, omnImodal reasoning, efficiency, and dataset/benchmark construction as key challenges and outlines future directions for advancing fully modal, explainable reasoning in real-world settings.

Abstract

With the remarkable success of Multimodal Large Language Models (MLLMs) in perception tasks, enhancing their complex reasoning capabilities has emerged as a critical research focus. Existing models still suffer from challenges such as opaque reasoning paths and insufficient generalization ability. Chain-of-Thought (CoT) reasoning, which has demonstrated significant efficacy in language models by enhancing reasoning transparency and output interpretability, holds promise for improving model reasoning capabilities when extended to the multimodal domain. This paper provides a systematic review centered on "Multimodal Chain-of-Thought" (MCoT). First, it analyzes the background and theoretical motivations for its inception from the perspectives of technical evolution and task demands. Then, it introduces mainstream MCoT methods from three aspects: CoT paradigms, the post-training stage, and the inference stage, while also analyzing their underlying mechanisms. Furthermore, the paper summarizes existing evaluation benchmarks and metrics, and discusses the application scenarios of MCoT. Finally, it analyzes the challenges currently facing MCoT and provides an outlook on its future research directions.

From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models

TL;DR

The paper surveys Multimodal Chain-of-Thought (MCoT) in Multimodal Large Language Models, detailing why explicit cross-modal reasoning improves performance and interpretability. It categorizes methods into CoT paradigms, post-training data/objectives, and inference-time techniques, and analyzes the underlying mechanisms of information representation, structured reasoning, and process supervision. It also catalogs evaluation benchmarks across math, spatiotemporal, vision-language transformation, sequential/multi-image, and integrated reasoning, and surveys applications in embodied AI, autonomous driving, healthcare, multimodal generation, translation, and social affective tasks. Finally, it discusses robustness, security, omnImodal reasoning, efficiency, and dataset/benchmark construction as key challenges and outlines future directions for advancing fully modal, explainable reasoning in real-world settings.

Abstract

With the remarkable success of Multimodal Large Language Models (MLLMs) in perception tasks, enhancing their complex reasoning capabilities has emerged as a critical research focus. Existing models still suffer from challenges such as opaque reasoning paths and insufficient generalization ability. Chain-of-Thought (CoT) reasoning, which has demonstrated significant efficacy in language models by enhancing reasoning transparency and output interpretability, holds promise for improving model reasoning capabilities when extended to the multimodal domain. This paper provides a systematic review centered on "Multimodal Chain-of-Thought" (MCoT). First, it analyzes the background and theoretical motivations for its inception from the perspectives of technical evolution and task demands. Then, it introduces mainstream MCoT methods from three aspects: CoT paradigms, the post-training stage, and the inference stage, while also analyzing their underlying mechanisms. Furthermore, the paper summarizes existing evaluation benchmarks and metrics, and discusses the application scenarios of MCoT. Finally, it analyzes the challenges currently facing MCoT and provides an outlook on its future research directions.

Paper Structure

This paper contains 42 sections, 4 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Survey Organization
  3. Background:What and Why CoT-MLLMs?
  4. Multimodal Large Language Model
  5. CoT and M-CoT
  6. Method: How CoT Empowers CoT-MLLMs?
  7. CoT Paradigm
  8. Post-Training Phase
  9. Training Data
  10. Training Paradigm
  11. Inference Phase
  12. CoT Prompting
  13. Search Strategies
  14. Self Refinement
  15. Knowledge Enhancement
  16. ...and 27 more sections

Figures (7)

  • Figure 1: The Organization of our survey.
  • Figure 2: Comparison between conventional MLLMs and naive M-CoT reasoning.
  • Figure 3: Comparison of reasoning paradigms among conventional MLLMs, text-only CoT, and M-CoT: (a) End-to-end “black-box” reasoning in conventional MLLMs; (b) Text-only CoT reasoning; (c) ICoT incorporates visual inputs as part of the reasoning content; (d) MVoT and Visual SKETCHPAD assist the reasoning process by generating images; (e) The BBA approach constructs separate reasoning chains for textual and visual modalities and performs alignment and integration across them.
  • Figure 4: Three training paradigms of CoT-MLLMs: (a) only supervised fine-tuning, (b) only reinforcement learning, (c) supervised fine-tuning + reinforcement learning
  • Figure 5: Summary of the inference-time scaling methods for CoT-MLLMs: (a) CoT prompting, (b) search strategies, (c) self refinement, (d) knowledge enhancement, (e) agent assistance
  • ...and 2 more figures