Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding

TL;DR

This work identifies a decoder bottleneck in vision-language models as a key reason for poor visual arithmetic, despite informative visual encoders. It introduces CogAlign, a Piaget-inspired post-training method using Direct Preference Optimization on synthetically generated, transformation-based tasks to enforce conservation and decentration. CogAlign yields significant improvements on probing tasks and transfers to downstream chart understanding and geometry problems, with greater data efficiency than task-specific fine-tuning. The results suggest strengthening foundational visual-arithmetic reasoning can broadly enhance multimodal generalization across tasks and models.

Abstract

Vision Language Models (VLMs) have achieved remarkable progress in multimodal tasks, yet they often struggle with visual arithmetic, seemingly simple capabilities like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning. In this work, we first investigate the root causes of this deficiency through a suite of probing tasks focusing on basic visual arithmetic. Our analysis reveals that while pre-trained vision encoders typically capture sufficient information, the text decoder often fails to decode it correctly for arithmetic reasoning. To address this, we propose CogAlign, a novel post-training strategy inspired by Piaget's theory of cognitive development. CogAlign trains VLMs to recognize invariant properties under visual transformations. We demonstrate that this approach significantly improves the performance of three diverse VLMs on our proposed probing tasks. Furthermore, CogAlign enhances performance by an average of 4.6% on CHOCOLATE and 2.9% on MATH-VISION, outperforming or matching supervised fine-tuning methods while requiring only 60% less training data. These results highlight the effectiveness and generalizability of CogAlign in improving fundamental visual arithmetic capabilities and their transfer to downstream tasks.

Figures (5)

• Figure 1: Examples of probing tasks designed to assess visual arithmetic abilities. Each task presents a visual input and a question requiring comparison or evaluation of geometric properties. At the bottom of each task, we see that even top-performing VLMs like GPT-4o and InternVL2.5-78B struggle with these seemly simple tasks.
• Figure 2: t-SNE visualizations of clustering by CLIP and SigLIP for angles and lines.
• Figure 3: Example training data for CogAlign. Each example consists of a visual input, a query prompting comparison of a specific property (i.e. angle, length, distance, and etc), a positive response consistent with the visual input, and a negative response that contradicts it.
• Figure 4: Performance comparison when training different models with SFT and DPO.
• Figure 5: Performance on two general VLM benchmarks: MME and MMMU with or without CogAlign.