AbductiveMLLM: Boosting Visual Abductive Reasoning Within MLLMs

TL;DR

This work tackles visual abductive reasoning by bridging verbal and pictorial cognition in multimodal large language models. It presents AbductiveMLLM, consisting of a Reasoner that generates and prunes causally relevant verbal hypotheses and an Imaginer that guides reasoning through a diffusion-based pictorial imagination with lightweight adapters. The two components are trained end-to-end, with a causality-aware contrastive learning objective for hypothesis selection and a latent diffusion loss to steer visual grounding. Empirical results on VAR and YouCookII show state-of-the-art performance, underscoring the value of integrating verbal hypotheses with pictorial imagining for Open-world video reasoning.

Abstract

Visual abductive reasoning (VAR) is a challenging task that requires AI systems to infer the most likely explanation for incomplete visual observations. While recent MLLMs develop strong general-purpose multimodal reasoning capabilities, they fall short in abductive inference, as compared to human beings. To bridge this gap, we draw inspiration from the interplay between verbal and pictorial abduction in human cognition, and propose to strengthen abduction of MLLMs by mimicking such dual-mode behavior. Concretely, we introduce AbductiveMLLM comprising of two synergistic components: REASONER and IMAGINER. The REASONER operates in the verbal domain. It first explores a broad space of possible explanations using a blind LLM and then prunes visually incongruent hypotheses based on cross-modal causal alignment. The remaining hypotheses are introduced into the MLLM as targeted priors, steering its reasoning toward causally coherent explanations. The IMAGINER, on the other hand, further guides MLLMs by emulating human-like pictorial thinking. It conditions a text-to-image diffusion model on both the input video and the REASONER's output embeddings to "imagine" plausible visual scenes that correspond to verbal explanation, thereby enriching MLLMs' contextual grounding. The two components are trained jointly in an end-to-end manner. Experiments on standard VAR benchmarks show that AbductiveMLLM achieves state-of-the-art performance, consistently outperforming traditional solutions and advanced MLLMs.

Figures (4)

• Figure 1: Network architecture of AbductiveMLLM. The network consists two synergistic components: Reasoner and Imaginer. On the left, Reasoner takes a query and a sequence of incomplete observations $\mathcal{V}$ as input. First, it generates captions $\mathcal{C}$ for each observed event. Based on $\mathcal{C}$, Causality-aware Hypotheses Generation module provides high-quality hypotheses $\mathcal{Y}$ for the MLLM. Its output embeddings are passed to Imaginer as textual conditions $\bm{c}_t$ for imagination, which are also used to generate verbal abduction results. On the right, Imaginer is adapted from a text-to-image diffusion model through the integration of lightweight adapters. It takes $\bm{c}_t$ and $\bm{c}_v$ as multimodal conditions, where $\bm{c}_v$ is visual local-global hybrid representations extracted from the observations. The two components are trained end-to-end with $\mathcal{L}_\text{CE}$ and $\mathcal{L}_\text{Diffusion}$.
• Figure 2: Illustration of cross-modal causal contrastive learning. Encoders $\Phi_V$, $\Phi_T$ learn to attract causally plausible hypotheses $\mathcal{P}^+$ to the observations, and repel causally irrelevant hypotheses $\mathcal{P}^-$.
• Figure 3: A standard U-Net block of Stable Diffusion with proposed adapters. During training, we only update parameters of the adapters (in yellow), and freeze parameters of other modules (in blue).
• Figure 4: Qualitative comparison of AbductiveMLLM on an example from VAR test.