LogicGaze: Benchmarking Causal Consistency in Visual Narratives via Counterfactual Verification

TL;DR

LogicGaze targets the problem of hallucination in vision-language models by testing whether sequential causal reasoning can be grounded in actual visual evidence. It introduces a three-stage causal representation $A\rightarrow B\rightarrow C$ together with visually grounded counterfactual perturbations, and builds a large, reproducible benchmark from 40,000 ShareGPT4Video clips and 5,000 Flickr30k images. The framework defines three tasks—Causal Validation, Grounded Narrative Synthesis, and Perturbation Rejection—to quantify grounding, narrative fidelity, and robustness, showing that state-of-the-art models struggle with grounded reasoning and outperforming baselines when using LogicGaze. The work provides strong evidence for the value of structured causal grounding in multimodal systems and releases all data and code in a anonymized repository to facilitate adoption and further research.

Abstract

While sequential reasoning enhances the capability of Vision-Language Models (VLMs) to execute complex multimodal tasks, their reliability in grounding these reasoning chains within actual visual evidence remains insufficiently explored. We introduce LogicGaze, a novel benchmark framework designed to rigorously interrogate whether VLMs can validate sequential causal chains against visual inputs, specifically targeting the pervasive issue of hallucination. Curated from 40,000 video segments from ShareGPT4Video and a subset of Flickr30k imagery, LogicGaze integrates causal sequences with visually contradictory yet linguistically plausible perturbations, compelling models to verify the authenticity of each reasoning step. Our tripartite evaluation protocol - Causal Validation, Grounded Narrative Synthesis, and Perturbation Rejection - exposes significant vulnerabilities in state-of-the-art VLMs such as Qwen2.5-VL-72B. LogicGaze advocates for robust, trustworthy multimodal reasoning, with all resources publicly available in an anonymized repository.

Figures (3)

• Figure 1: Statistical Distribution of LogicGaze Datasets. Video (ShareGPT4Video): 40,000 samples, avg. sequence length 5.1 events. Image (Flickr30k): 5,000 samples, avg. 3 causal steps (A$\to$B$\to$C).
• Figure 2: Efficiency benchmarking on PopQA using the Qwen2.5-7B backbone. LogicGaze achieves superior accuracy with minimal latency and token consumption. Best results are highlighted in bold.
• Figure 3: Ablation analysis on retrieval depth ($k$) and contrastive weight ($\lambda$). Optimal configuration is highlighted.