Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning

Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, Godawari Sudhakar Rao

TL;DR

KAM-CoT addresses the challenge of scalable multimodal reasoning by integrating chain-of-thought with knowledge-graph grounding. It employs a two-stage process that first generates rationale tokens and then infers answers, grounding reasoning in a ConceptNet-derived subgraph and fusing language, vision, and graph representations via cross-attention and gated fusion. The approach achieves state-of-the-art results on ScienceQA (accuracy $93.87\%$ with only $\sim$280M parameters) and demonstrates robustness across modalities and data regimes, including favorable ablations with captions and node-count variations. This work highlights the practical potential of knowledge-grounded multimodal CoT for resource-efficient, reliable reasoning in AI systems. It also points to future work in extending KG grounding to domain-specific knowledge and scaling to larger model families.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance in natural language processing tasks by leveraging chain of thought (CoT) that enables step-by-step thinking. Extending LLMs with multimodal capabilities is the recent interest, but incurs computational cost and requires substantial hardware resources. To address these challenges, we propose KAM-CoT a framework that integrates CoT reasoning, Knowledge Graphs (KGs), and multiple modalities for a comprehensive understanding of multimodal tasks. KAM-CoT adopts a two-stage training process with KG grounding to generate effective rationales and answers. By incorporating external knowledge from KGs during reasoning, the model gains a deeper contextual understanding reducing hallucinations and enhancing the quality of answers. This knowledge-augmented CoT reasoning empowers the model to handle questions requiring external context, providing more informed answers. Experimental findings show KAM-CoT outperforms the state-of-the-art methods. On the ScienceQA dataset, we achieve an average accuracy of 93.87%, surpassing GPT-3.5 (75.17%) by 18% and GPT-4 (83.99%) by 10%. Remarkably, KAM-CoT achieves these results with only 280M trainable parameters at a time, demonstrating its cost-efficiency and effectiveness.

KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning

TL;DR

KAM-CoT addresses the challenge of scalable multimodal reasoning by integrating chain-of-thought with knowledge-graph grounding. It employs a two-stage process that first generates rationale tokens and then infers answers, grounding reasoning in a ConceptNet-derived subgraph and fusing language, vision, and graph representations via cross-attention and gated fusion. The approach achieves state-of-the-art results on ScienceQA (accuracy with only 280M parameters) and demonstrates robustness across modalities and data regimes, including favorable ablations with captions and node-count variations. This work highlights the practical potential of knowledge-grounded multimodal CoT for resource-efficient, reliable reasoning in AI systems. It also points to future work in extending KG grounding to domain-specific knowledge and scaling to larger model families.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance in natural language processing tasks by leveraging chain of thought (CoT) that enables step-by-step thinking. Extending LLMs with multimodal capabilities is the recent interest, but incurs computational cost and requires substantial hardware resources. To address these challenges, we propose KAM-CoT a framework that integrates CoT reasoning, Knowledge Graphs (KGs), and multiple modalities for a comprehensive understanding of multimodal tasks. KAM-CoT adopts a two-stage training process with KG grounding to generate effective rationales and answers. By incorporating external knowledge from KGs during reasoning, the model gains a deeper contextual understanding reducing hallucinations and enhancing the quality of answers. This knowledge-augmented CoT reasoning empowers the model to handle questions requiring external context, providing more informed answers. Experimental findings show KAM-CoT outperforms the state-of-the-art methods. On the ScienceQA dataset, we achieve an average accuracy of 93.87%, surpassing GPT-3.5 (75.17%) by 18% and GPT-4 (83.99%) by 10%. Remarkably, KAM-CoT achieves these results with only 280M trainable parameters at a time, demonstrating its cost-efficiency and effectiveness.
Paper Structure (37 sections, 12 equations, 7 figures, 9 tables, 1 algorithm)

This paper contains 37 sections, 12 equations, 7 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. In-context learning
  4. CoT through fine-tuning approaches
  5. Vision-Language Models
  6. Knowledge augmented methods
  7. Method
  8. Task Formulation
  9. Encode Inputs From Different Modalities
  10. Text Encoding
  11. Image Encoding
  12. Subgraph Extraction
  13. Graph Encoding
  14. Interaction Between Modalities
  15. Fusion
  16. ...and 22 more sections

Figures (7)

  • Figure 1: An example from ScienceQA dataset lu2022learn showing how graphs can aid in multi-modal QA.
  • Figure 2: KAM-CoT model architecture.
  • Figure 3: Performance of the fusion mechanisms on the validation set, evaluated using T5-Base.
  • Figure 4: Comparative performance using subsets of training data with MM-CoTFLAN-T5-Base (100% training data, zhang2023multimodal), and the human average.
  • Figure 5: A pictorial representation of batches with graph data. Here, the $3 \times 3$ blocks in the top-left, the $5 \times 5$ blocks around the center and the lower-right $2 \times 2$ blocks are 3 disjoint graphs. The connections between nodes are indicated by the intensity of fill in corresponding cell of the adjacency matrix. Note that three nodes in the first graph do not have any relation with any node beyond themselves. A similar observation can be made in the other two graphs as well. Although disjoint, they can be put into a single adjacency matrix as depicted above.
  • ...and 2 more figures