DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

TL;DR

DDCoT introduces Duty-Distinct Chain-of-Thought prompting to enable robust, explainable multimodal reasoning in language models. It splits the reasoning and visual recognition tasks, uses negative-space prompting to signal uncertainty, and leverages a VQA model to supply visual information before joint reasoning with LLMs. The approach yields state-of-the-art results on ScienceQA in both zero-shot and fine-tuning settings and demonstrates strong generalization with ablations validating the importance of its components (DLP, RCVE) and uncertainty signaling. This work reduces annotation needs, improves interpretability, and enhances robustness in multimodal reasoning systems.

Abstract

A long-standing goal of AI systems is to perform complex multimodal reasoning like humans. Recently, large language models (LLMs) have made remarkable strides in such multi-step reasoning on the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. However, the transfer of these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation and the limitations in terms of flexibility, generalizability, and explainability. To evoke CoT reasoning in multimodality, this work first conducts an in-depth analysis of these challenges posed by multimodality and presents two key insights: "keeping critical thinking" and "letting everyone do their jobs" in multimodal CoT reasoning. Furthermore, this study proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process. The rationales generated by DDCoT not only improve the reasoning abilities of both large and small language models in zero-shot prompting and fine-tuning learning, significantly outperforming state-of-the-art methods but also exhibit impressive generalizability and explainability.

Figures (13)

• Figure 1: Comparison of existing multimodal CoT methods and our DDCoT on (a), (c), (d) and (e) an out-of-distribution example to illustrate the generalizability and (b) performance of zero-shot and fine-tuning learning, demonstrating that we are the first to generate general multimodal rationales.
• Figure 2: An example shows the importance of input rationales in multimodal reasoning as well as the disparate roles of rationales in zero-shot and fine-tuning scenarios.
• Figure 3: An example for disparate roles of rationales in zero-shot and fine-tuning scenarios, with question depicted in Figure \ref{['fig:intro']}.
• Figure 4: An example of the hallucination challenge which is exacerbated by providing interleaved multimodal information at once in generating rationales with LLMs.
• Figure 5: An overview of our DDCoT and its utilization to improve the multimodal reasoning of LMs. Note that although errors encounter in the second sub-problem during visual recognition, the language model rectifies this error in the joint reasoning step with critical thought.