Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models

TL;DR

This work addresses the limited perceptual and reasoning capabilities of large Vision-Language Models by introducing AGILE, an agentic jigsaw interaction learning framework that treats jigsaw solving as a stepwise, interactive process. The model generates executable Python actions to manipulate a visual environment, receiving fine-grained feedback that drives progressive improvements in perception and reasoning. A scalable, code-based data-generation pipeline (including a cold-start set of 1.6K trajectories and 15.6K RL images) enables efficient reinforcement learning with GRPO, yielding dramatic gains on $2 \times 2$ jigsaws ($9.5\% \rightarrow 82.8\%$ accuracy) and substantial generalization across nine downstream vision tasks (average $+3.1\%$). The results demonstrate the viability of jigsaw-based proxy tasks to alleviate multimodal RL data scarcity and enhance cross-task visual perception and reasoning, suggesting a practical, scalable path for improved multimodal understanding.

Abstract

Although current large Vision-Language Models (VLMs) have advanced in multimodal understanding and reasoning, their fundamental perceptual and reasoning abilities remain limited. Specifically, even on simple jigsaw tasks, existing VLMs perform near randomly, revealing deficiencies in core perception and reasoning capabilities. While high-quality vision-language data can enhance these capabilities, its scarcity and limited scalability impose significant constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction Learning for Enhancing visual perception and reasoning in VLMs. AGILE formulates jigsaw solving as an interactive process, enabling the model to progressively engage with the environment. At each step, the model generates executable code to perform an action based on the current state, while the environment provides fine-grained visual feedback to guide task completion. Through this iterative cycle of observation and interaction, the model incrementally improves its perceptual and reasoning capabilities via exploration and feedback. Experimental results show that AGILE not only substantially boosts performance on jigsaw tasks of varying complexity (e.g., increasing accuracy from 9.5% to 82.8% under the 2 $\times$ 2 setting) but also demonstrates strong generalization across 9 general vision tasks, achieving an average improvement of 3.1%. These results indicate notable enhancements in both perceptual and reasoning abilities. This work opens a new avenue for advancing reasoning and generalization in multimodal models and provides an efficient, scalable solution to the scarcity of multimodal reinforcement learning data. The code and datasets is available at https://github.com/yuzeng0-0/AGILE .

Figures (11)

• Figure 1: Description of the action space. (a) illustrates swapping two jigsaw pieces and observing the updated jigsaw state; (b) shows cropping a specific region of the jigsaw for closer inspection; and (c) depicts zooming into a selected area to examine fine-grained details.
• Figure 2: Overview of the AGILE framework. (a) depicts the interaction process between the model and the external environment, together with the implementation of the GRPO algorithm; (b) shows the collection of high-quality jigsaw trajectory data; and (c) illustrates the model–environment interaction during the jigsaw rollout process.
• Figure 3: Impact of Training Data Scale. The left y-axis denotes the accuracies on HRBench4K and RealWorldQA, while the right y-axis corresponds to the accuracy on the jigsaw task.
• Figure 4: Comparison with General QA Data. The total number of samples is consistently maintained at 20K across both experimental setups.
• Figure 5: Case Study. Jigsaw-solving reasoning and behaviors exhibited by our model.