Can Multimodal Large Language Model Think Analogically?

TL;DR

A unified prompt template and a method for harnessing the comprehension capabilities of MLLM to augment existing models are proposed and outperforms existing methods on popular datasets, providing preliminary evidence for the analogical reasoning capability of MLLM.

Abstract

Analogical reasoning, particularly in multimodal contexts, is the foundation of human perception and creativity. Multimodal Large Language Model (MLLM) has recently sparked considerable discussion due to its emergent capabilities. In this paper, we delve into the multimodal analogical reasoning capability of MLLM. Specifically, we explore two facets: \textit{MLLM as an explainer} and \textit{MLLM as a predictor}. In \textit{MLLM as an explainer}, we primarily focus on whether MLLM can deeply comprehend multimodal analogical reasoning problems. We propose a unified prompt template and a method for harnessing the comprehension capabilities of MLLM to augment existing models. In \textit{MLLM as a predictor}, we aim to determine whether MLLM can directly solve multimodal analogical reasoning problems. The experiments show that our approach outperforms existing methods on popular datasets, providing preliminary evidence for the analogical reasoning capability of MLLM.

Figures (8)

• Figure 1: Humans often establish initial cognitive understanding through multimodal analogical reasoning when dealing with unfamiliar problems. One analogical reasoning example is $Sun : Solar\;system :: Nucleus : Atom.$
• Figure 2: Overview of the multimodal analogical reasoning task. We provide a multimodal knowledge graph in Figure \ref{['fig:def']}(a). Figure \ref{['fig:def']}(b) shows a general analogical reasoning task. Figure \ref{['fig:def']}(c) shows three subtasks of multimodal analogical reasoning task. Please note that the relations indicated by dashed arrows ($\dashrightarrow$) and the text in parentheses beneath the images are for annotation purposes only and are not included in the input.
• Figure 3: MLLM as an explainer.
• Figure 4: MLLM as a predictor.
• Figure 5: Q&A and multiple-choice evaluation. * denotes Predictor.