Evaluating the encoding competence of visual language models using uncommon actions

TL;DR

This work introduces UAIT, a benchmark dataset and evaluation framework designed to probe visual language models on uncommon-sense action understanding by generating grammatically valid yet semantically counterintuitive image-text pairs. It combines VerbNet-guided verb selection, few-shot large language model prompts, and Stable Diffusion-based image synthesis to create a controlled, bias-reduced diagnostic platform with two-option VQA questions. Across multiple VLMs and contrastive baselines, humans significantly outperform models, even when chain-of-thought prompting or LoRA fine-tuning is applied, revealing persistent gaps in semantic-role reasoning and physical feasibility understanding. The study provides concrete insights and a roadmap for improving multimodal reasoning through enhanced semantic-role modeling, anti-bias training, and a causal-reasoning oriented learning paradigm that moves beyond surface-pattern matching.

Abstract

We propose UAIT (Uncommon-sense Action Image-Text) dataset, a new evaluation benchmark designed to test the semantic understanding ability of visual language models (VLMs) in uncommon-sense action scenes. Unlike previous datasets that focus on common visual scenes with statistical frequency advantages, UAIT challenges models with grammatically reasonable but semantically counter-common sense image-text pairs. Such tasks require models to go beyond superficial pattern recognition and demonstrate a deep understanding of agent-patient relationships and physical feasibility. To build UAIT, we designed a semi-automated process to synthesize high-quality uncommon-sense image-text samples using large language models, few-shot prompt engineering, and text-to-image generation. Each sample is accompanied by a carefully designed multiple-choice question to test the model's competence in fine-grained reasoning. We evaluate multiple state-of-the-art visual language models and compare them with models based on contrastive learning. Experiments show that all models perform significantly worse than humans in semantic judgment, especially in distinguishing grammatical correctness from semantic rationality. Further experiments show that even the lightweight model can improve its accuracy after fine-tuning, demonstrating the great potential of directional adaptation. This study not only reveals the key weaknesses of VLMs, but also provides diagnostic tools and research directions for the development of robust models with real visual semantic reasoning capabilities.

Figures (6)

• Figure 1: This is an example from UAIT, the correct description of the action scene is “a rabbit is dragging a tiger”. Below are the reasons why the image is beyond the common-sense understanding.
• Figure 2: The construction framework of uncommon image-text
• Figure 3: examples of a VQA tasks in the UAIT dataset. Each image corresponds to a question and two options. To eliminate the influence of CoT when the model answers, the end of the question contains "Output only the corresponding letter of the option without providing a reason."
• Figure 4: An example of model answer on the VQA task in UAIT dataset, we use qwen2-vl-instruct as a base model here.
• Figure 5: lora finetuning loss of LLaVA-1.5 on UAIT