Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models

TL;DR

This paper introduces AVSD, a dataset that decomposes basic 2D Euclidean geometry perception into 36 atomic visual skills to probe Vision-Language Models. By evaluating a wide range of models on AVSD and a nu-geometry failure benchmark, it shows that state-of-the-art VLMs struggle with atomic geometric perception, with limited gains from chain-of-thought prompting and notable sensitivity to image style. AVSD comprises handcrafted (AVSD-h), synthetic (AVSD-s), and style-augmented (AVSD-c) subdatasets, totaling 13,188 problems across 36 skills, designed to enable skill isolation and robust evaluation. The study demonstrates that targeted pre-training on atomic visual skills, rather than solely composite geometry data, can improve performance and generalization, signaling a shift toward atomic-skill-centric training for geometric perception. These findings highlight the need for purpose-built, atomic-skill datasets to advance robust multimodal reasoning in geometry and related scientific visual tasks, with ν-geometry proposed as a broader evaluation framework for future work.

Abstract

Recent Vision-Language Models (VLMs) have demonstrated impressive multimodal comprehension and reasoning capabilities, yet they often struggle with trivially simple visual tasks. In this work, we focus on the domain of basic 2D Euclidean geometry and systematically categorize the fundamental, indivisible visual perception skills, which we refer to as atomic visual skills. We then introduce the Atomic Visual Skills Dataset (AVSD) for evaluating VLMs on the atomic visual skills. Using AVSD, we benchmark state-of-the-art VLMs and find that they struggle with these tasks, despite being trivial for adult humans. Our findings highlight the need for purpose-built datasets to train and evaluate VLMs on atomic, rather than composite, visual perception tasks.

Figures (16)

• Figure 1: Examples of AVSD problems and responses by o3 model. Other state-of-the-art models exhibit similar failures. These examples demonstrate a deficiency in the VLMs' understanding of basic geometric concepts.
• Figure 2: Examples of $\nu$-geometry. These tasks test composite geometric perception but do not require any mathematical reasoning. They demonstrate that the state-of-the-art VLMs struggle with geometric perception, even before they get to geometric reasoning.
• Figure 3: List of 36 atomic visual skills and the number of easy, medium, and hard problems for each skill from AVSD-h. The difficulty is judged by the authors. We provide a total of 5,163 new handcrafted problems.
• Figure 4: Statistics of GPT-4o response on the same question with different styles. This example shows that VLMs are sensitive to the variation in image style. This motivates the AVSD-c sub-dataset, designed to assess VLMs' robustness to perceive geometric features independent of image style.
• Figure 5: AVSD-c consists of synthetically generated images with diverse styles imbued with ControlNet. The generation content is conditioned on the Canny edges of the input image while the style is conditioned on the natural language prompt.