[De|Re]constructing VLMs' Reasoning in Counting

TL;DR

This work interrogates the counting abilities of seven state-of-the-art Vision-Language Models, moving beyond score-based benchmarks to diagnose the sources of counting errors. Through a CIVET-based framework, it uses balanced synthetic stimuli and layer-wise probing to show that the primary bottleneck lies in the final output mapping, not the encoder representations. A targeted approach that fine-tunes only the output layer yields substantial gains (up to 21% on synthetic data and notable transfer to real-world data), validating the method across both synthetic and real datasets. The study highlights persistent pretraining biases and offers a practical, scalable remedy, with implications for improving reasoning in VLMs and for designing more robust, unbiased evaluation protocols.

Abstract

Vision-Language Models (VLMs) have recently gained attention due to their competitive performance on multiple downstream tasks, achieved by following user-input instructions. However, VLMs still exhibit several limitations in visual reasoning, such as difficulties in identifying relations (e.g., spatial, temporal, and among objects), understanding temporal sequences (e.g., frames), and counting objects. In this work, we go beyond score-level benchmark evaluations of VLMs by investigating the underlying causes of their failures and proposing a targeted approach to improve their reasoning capabilities. We study the reasoning skills of seven state-of-the-art VLMs in the counting task under controlled experimental conditions. Our experiments show that VLMs are highly sensitive to the number and type of objects, their spatial arrangement, and the co-occurrence of distractors. A layer-wise analysis reveals that errors are due to incorrect mapping of the last-layer representation into the output space. Our targeted training shows that fine-tuning just the output layer improves accuracy by up to 21%. We corroborate these findings by achieving consistent improvements on real-world datasets.

Figures (10)

• Figure 1: Example images from each experimental setting used to evaluate the counting performance of VLMs under different conditions. In all cases, models are asked to answer the question: “How many magenta circles are there?”. The Baseline setting includes only target objects. The Distractors setting introduces irrelevant objects to assess their impact on model accuracy. The Clustered vs. Scattered setting examines whether specific spatial arrangements of objects affect performance. For clarity, distractors have been depicted in a different color to enhance the contrast with the targets.
• Figure 2: F1-scores under the baseline setting for CLIP, CoCa, and the Encoder-Decoder models when increasing the number of target objects. Results for Encoder-Decoder models are averaged across LLaVA-Interleave, LLaVA-OneVision, PaliGemma 2, and Qwen2-VL, with the corresponding standard deviation reported using error bars.
• Figure 3: F1-scores under the baseline setting for different numbers of target objects when querying encoder-decoder models with open- or closed-ended questions. Results are averaged across encoder-decoder models with the corresponding standard deviation indicated using error bars.
• Figure 4: Layer-wise analysis of LLaVA-OneVision under the baseline setting. For each layer $l_i$, we report the counting accuracy of the probing model $P_i$ when trained on different intermediate representations, as discussed in Section \ref{['subsec:identifying']}. Results show that the output layer is the main source of errors, since training $P_i$ on the last token representation $H_{\text{last}}$ yields near-perfect accuracy.
• Figure 5: Change in F1-Score (%) and Mean Absolute Error (MAE) after fine-tuning on synthetic (left) and real-world (right) data as a function of the number of target objects. The number of target objects ranges from 1 to 9 in the synthetic setting and from 2 to 9 in the real-world setting. Bars indicate average improvement across encoder-decoder models, with error bars showing standard deviation. The shaded regions represent the maximum possible improvement relative to the performance of the pre-trained models.