Revealing Multimodal Causality with Large Language Models

TL;DR

This work tackles causal discovery from multimodal unstructured data by integrating multimodal foundation models and large language models into a three-component framework: contrastive factor discovery to identify causal variables across modalities, statistical causal structure discovery to infer relationships among them, and multimodal counterfactual reasoning to iteratively refine the results using world knowledge. The method leverages CLIP-style representations, the FCI algorithm for structure learning, and MLLM-driven counterfactuals with semantic and causal checks to reduce structural ambiguity. Empirical results on synthetic MAG and real-world Lung Cancer datasets demonstrate improved factor identification and more accurate causal graphs compared with baselines, with ablations confirming the complementary roles of CFD and MCR. The approach advances practical multimodal causal discovery, while outlining limitations around dataset scale, modality coverage, and potential MLLM biases, and suggesting directions for robust validation and broader modality support.

Abstract

Uncovering cause-and-effect mechanisms from data is fundamental to scientific progress. While large language models (LLMs) show promise for enhancing causal discovery (CD) from unstructured data, their application to the increasingly prevalent multimodal setting remains a critical challenge. Even with the advent of multimodal LLMs (MLLMs), their efficacy in multimodal CD is hindered by two primary limitations: (1) difficulty in exploring intra- and inter-modal interactions for comprehensive causal variable identification; and (2) insufficiency to handle structural ambiguities with purely observational data. To address these challenges, we propose MLLM-CD, a novel framework for multimodal causal discovery from unstructured data. It consists of three key components: (1) a novel contrastive factor discovery module to identify genuine multimodal factors based on the interactions explored from contrastive sample pairs; (2) a statistical causal structure discovery module to infer causal relationships among discovered factors; and (3) an iterative multimodal counterfactual reasoning module to refine the discovery outcomes iteratively by incorporating the world knowledge and reasoning capabilities of MLLMs. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed MLLM-CD in revealing genuine factors and causal relationships among them from multimodal unstructured data.

Figures (23)

• Figure 1: Illustration of MLLM-CD in lung cancer diagnosis. It first employs contrastive factor discovery with MLLMs to identify potential causal variables and form structured data. Then, a CD algorithm is performed to infer causal structures. To further reduce ambiguities, it leverages MLLM's world knowledge to generate multimodal counterfactual samples for iterative refinement.
• Figure 2: Results on MAG dataset1. (a) Ground truth. (b) Causal graph discovered by COAT. Visual and verbal causal variables are presented in blue and black.
• Figure 3: Discovered graphs on MAG dataset.
• Figure 4: Discovered causal graphs on Lung Cancer dataset.
• Figure 5: Ground truth and faithful (via FCI algorithm) causal graphs in the MAG dataset.