From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models

TL;DR

This survey offers a nuanced exposition of current methodologies applied in CMR using LLMs, classifying these into a detailed three-tiered taxonomy and delves into the principal design strategies and operational techniques of prototypical models within this domain.

Abstract

Cross-modal reasoning (CMR), the intricate process of synthesizing and drawing inferences across divergent sensory modalities, is increasingly recognized as a crucial capability in the progression toward more sophisticated and anthropomorphic artificial intelligence systems. Large Language Models (LLMs) represent a class of AI algorithms specifically engineered to parse, produce, and engage with human language on an extensive scale. The recent trend of deploying LLMs to tackle CMR tasks has marked a new mainstream of approaches for enhancing their effectiveness. This survey offers a nuanced exposition of current methodologies applied in CMR using LLMs, classifying these into a detailed three-tiered taxonomy. Moreover, the survey delves into the principal design strategies and operational techniques of prototypical models within this domain. Additionally, it articulates the prevailing challenges associated with the integration of LLMs in CMR and identifies prospective research directions. To sum up, this survey endeavors to expedite progress within this burgeoning field by endowing scholars with a holistic and detailed vista, showcasing the vanguard of current research whilst pinpointing potential avenues for advancement. An associated GitHub repository that collects the relevant papers can be found at https://github.com/ZuyiZhou/Awesome-Cross-modal-Reasoning-with-LLMs

Figures (11)

• Figure 1: The taxonomy of the roles of LLMs in cross-modal reasoning.
• Figure 2: Examples of Cross-Modal Reasoning utilizing Large Language Models (LLMs). The illustrated scenarios highlight the potent capabilities of LLMs in facilitating effective cross-modal reasoning and aiding in comprehension across various modalities.
• Figure 3: The multifaceted role of Large Language Models (LLMs) within the domain of Cross-Modal Reasoning (CMR): (a) LLMs as multimodal fusion engines enact a pivotal function in the alignment, fusion, and integration of multimodal inputs into coherent textual representations. These multimodal inputs necessitate a series of transformations to render them compatible with the intricate architecture of LLMs. Subsequently, the processed data is relayed to auxiliary output modules, culminating in the generation of responses. (b) LLMs as textual processors analyze different types of textual tokens and generate appropriate texts to fulfill the requirements of other modules. (c) LLMs as cognitive controllers exercise a critical evaluative role in discerning the task-specific requirements and appraising the practical feasibility of potential implementations, thereby orchestrating the reasoning methodology. The reasoning sequences produced by LLMs are subsequently operationalized by supplementary modules in the system. (d) LLMs as knowledge enhancers can provide valuable support to CMR tasks by offering a wealth of knowledge derived not just from their large datasets but also from external sources and real-time contextual information.
• Figure 4: Comprehensive overview of Cross-Modal Reasoning with Large Language Models (CMR with LLMs) categories and corresponding methodologies.
• Figure 5: The function of Large Language Models (LLMs) as Multimodal Fusion Engine within cross-modal reasoning frameworks.