VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning

TL;DR

The paper tackles Abstract Visual Reasoning (AVR) bottlenecks in Multimodal LLMs by identifying fine-grained perception as the key limitation. It introduces VisuRiddles, a benchmark for AVR across multiple dimensions, and VisuRiddles Synthesizer to generate perceptually annotated training data, enabling supervision of perceptual processes. Building on these, the authors propose Perception-Augmented Visual Reasoner (PAVR), a two-stage framework combining supervised fine-tuning on perceptual descriptions with reinforcement-learning–based reasoning to improve AVR. Experimental results show that perception-grounded training significantly boosts AVR performance, while traditional scaling and CoT approaches alone do not close the gap, highlighting the practical impact of integrating fine-grained perception with reasoning for abstract visual tasks.

Abstract

Recent strides in multimodal large language models (MLLMs) have significantly advanced their performance in many reasoning tasks. However, Abstract Visual Reasoning (AVR) remains a critical challenge, primarily due to limitations in perceiving abstract graphics. To tackle this issue, we investigate the bottlenecks in current MLLMs and synthesize training data to improve their abstract visual perception. First, we propose VisuRiddles, a benchmark for AVR, featuring tasks meticulously constructed to assess models' reasoning capacities across five core dimensions and two high-level reasoning categories. Second, we introduce the Perceptual Riddle Synthesizer (PRS), an automated framework for generating riddles with fine-grained perceptual descriptions. PRS not only generates valuable training data for abstract graphics but also provides fine-grained perceptual description, crucially allowing for supervision over intermediate reasoning stages and thereby improving both training efficacy and model interpretability. Our extensive experimental results on VisuRiddles empirically validate that fine-grained visual perception is the principal bottleneck and our synthesis framework markedly enhances the performance of contemporary MLLMs on these challenging tasks. Our code and dataset will be released at https://github.com/yh-hust/VisuRiddles

Figures (7)

• Figure 1: Performance and analysis of MLLMs on AVR. (a) Most advanced MLLMs achieve limited accuracy on VisuRiddles, often close to random choice and far below human performance. (b) Model responses to abstract graphics and their perceptual descriptions show that once equipped with perceptual capability, MLLMs can succeed in AVR task.
• Figure 2: Representative examples from VisuRiddles. The benchmark includes eight reasoning categories, designed to comprehensively evaluate diverse reasoning capabilities of MLLMs.
• Figure 3: Overview of the VisuRiddles Synthesizer. (a) A unified pipeline for generating abstract graphics with fine-grained perceptual descriptions. (b) Visualization of synthesized riddles based on positional rule and stylistic rule.
• Figure 4: Overview of PAVR. (i) Baseline: Incorrect perception leads to incorrect results. (ii) PAVR-SFT, which is trained on synthesized data to enhance fine-grained perceptual ability, can accurately understand visual content in riddles but fails in pattern identification. (iii) PAVR, which builds upon PAVR-SFT with reinforcement learning, can effectively recognize patterns and derive the correct answer.
• Figure 5: Case study comparing different reasoning strategies on a VisuRiddles example. (a) QVQ-72B reflects a flawed loop under the "thinking" mode. (b) Qwen2.5-VL-72B with CoT prompting exhibits incorrect perceptual understanding. (c) PAVR exhibits accurate perception and coherent reasoning, ultimately arriving at the correct answer.