Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning

Zhaotian Weng, Haoxuan Li, Xin Eric Wang, Kuan-Hao Huang, Jieyu Zhao

TL;DR

Vision-language models demonstrate strong object and activity recognition but falter on causal order reasoning. The paper introduces VQA-Causal and VCR-Causal to isolate causal understanding via caption pairs that differ only in causal order, revealing near-random performance across diverse models. A data-level analysis shows causal expressions are extremely scarce in pretraining data, motivating a hard-negative fine-tuning strategy (CausalCLIP) that improves causal reasoning while preserving downstream tasks. The work underscores a core gap in multimodal causal understanding and provides a practical, data-centric path toward bridging it, with implications for future architectural and dataset design.

Abstract

Despite the impressive performance of vision-language models (VLMs) on downstream tasks, their ability to understand and reason about causal relationships in visual inputs remains unclear. Robust causal reasoning is fundamental to solving complex high-level reasoning tasks, yet existing benchmarks often include a mixture of reasoning questions, and VLMs can frequently exploit object recognition and activity identification as shortcuts to arrive at the correct answers, making it challenging to truly assess their causal reasoning abilities. To bridge this gap, we introduce VQA-Causal and VCR-Causal, two new benchmarks specifically designed to isolate and rigorously evaluate VLMs' causal reasoning abilities. Our findings reveal that while VLMs excel in object and activity recognition, they perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. Further analysis suggests that this limitation stems from a severe lack of causal expressions in widely used training datasets, where causal relationships are rarely explicitly conveyed. We additionally explore fine-tuning strategies with hard negative cases, showing that targeted fine-tuning can improve model's causal reasoning while maintaining generalization and downstream performance. Our study highlights a key gap in current VLMs and lays the groundwork for future work on causal understanding.

What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning

TL;DR

Vision-language models demonstrate strong object and activity recognition but falter on causal order reasoning. The paper introduces VQA-Causal and VCR-Causal to isolate causal understanding via caption pairs that differ only in causal order, revealing near-random performance across diverse models. A data-level analysis shows causal expressions are extremely scarce in pretraining data, motivating a hard-negative fine-tuning strategy (CausalCLIP) that improves causal reasoning while preserving downstream tasks. The work underscores a core gap in multimodal causal understanding and provides a practical, data-centric path toward bridging it, with implications for future architectural and dataset design.

Abstract

Despite the impressive performance of vision-language models (VLMs) on downstream tasks, their ability to understand and reason about causal relationships in visual inputs remains unclear. Robust causal reasoning is fundamental to solving complex high-level reasoning tasks, yet existing benchmarks often include a mixture of reasoning questions, and VLMs can frequently exploit object recognition and activity identification as shortcuts to arrive at the correct answers, making it challenging to truly assess their causal reasoning abilities. To bridge this gap, we introduce VQA-Causal and VCR-Causal, two new benchmarks specifically designed to isolate and rigorously evaluate VLMs' causal reasoning abilities. Our findings reveal that while VLMs excel in object and activity recognition, they perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. Further analysis suggests that this limitation stems from a severe lack of causal expressions in widely used training datasets, where causal relationships are rarely explicitly conveyed. We additionally explore fine-tuning strategies with hard negative cases, showing that targeted fine-tuning can improve model's causal reasoning while maintaining generalization and downstream performance. Our study highlights a key gap in current VLMs and lays the groundwork for future work on causal understanding.

Paper Structure

This paper contains 12 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Benchmarks for Causal Order Reasoning
  3. Dataset Construction
  4. Causal Order Reasoning Test
  5. Benchmarking Results
  6. Activity and Object Understanding Test
  7. Why Struggling with Causal Reasoning? A Data-Level Exploration
  8. Data-Level Improvement
  9. Related Work
  10. Conclusion
  11. Limitations
  12. Appendix

Figures (2)

  • Figure 1: Examples from the VQA-Causal test and the Object and Activity Understanding test. Models tend to focus on low-level visual features such as objects and activities which are represented by the red and blue nodes in the scene graph on the left, but fail to capture high-level visual features such as relationships between activities, especially causal relationships in our case, which are represented by the green nodes in the scene graph.
  • Figure 2: The VCR dataset fails to genuinely evaluate a model's causal reasoning ability. In this example, the model can eliminate choice 3 by recognizing that there is no train tracks in the image. It can also rule out captions 2 and 1 by observing that the person is not tying the rope to or climb over anything. As a result, the model can arrive at the correct answer purely through object and activity understanding, without requiring genuine causal reasoning.