Visual Jigsaw Post-Training Improves MLLMs

TL;DR

Visual Jigsaw introduces a self-supervised post-training task that improves vision-centric understanding in multimodal LLMs without modifying architecture. It frames visual understanding as a permutation-ordering problem across images, videos, and 3D data, optimized via GRPO within the RLVR framework and requiring no extra generative components. Across three modalities, it yields consistent gains in fine-grained perception, temporal reasoning, and 3D spatial understanding, outperforming several strong baselines. The approach demonstrates the value of vision-centered pretext tasks for robust multimodal reasoning and motivates further research into perception-focused self-supervision for MLLMs.

Abstract

Reinforcement learning based post-training has recently emerged as a powerful paradigm for enhancing the alignment and reasoning capabilities of multimodal large language models (MLLMs). While vision-centric post-training is crucial for enhancing MLLMs' intrinsic understanding of visual signals, current post-training paradigms are predominantly text-centric, where dense visual inputs are only leveraged to extract sparse cues for text-based reasoning. There exist a few approaches in this direction, however, they often still rely on text as an intermediate mediator or introduce additional visual generative designs. In this work, we introduce Visual Jigsaw, a generic self-supervised post-training framework designed to strengthen visual understanding in MLLMs. Visual Jigsaw is formulated as a general ordering task: visual inputs are partitioned, shuffled, and the model must reconstruct the visual information by producing the correct permutation in natural language. This naturally aligns with reinforcement learning from verifiable rewards (RLVR), requires no additional visual generative components, and derives its supervisory signal automatically without any annotations. We instantiate Visual Jigsaw across three visual modalities, including images, videos, and 3D data. Extensive experiments demonstrate substantial improvements in fine-grained perception, temporal reasoning, and 3D spatial understanding. Our findings highlight the potential of self-supervised vision-centric tasks in post-training MLLMs and aim to inspire further research on vision-centric pretext designs. Project Page: https://penghao-wu.github.io/visual_jigsaw/

Figures (9)

• Figure 1: We propose Visual Jigsaw, a self-supervised post-training task that enhances visual perception and understanding in MLLMs. Training on visual jigsaw tasks substantially strengthens fine-grained perception, monocular spatial perception, and compositional visual understanding in images; temporal understanding in videos; and geometry-aware understanding in 3D, demonstrating its generality and effectiveness across modalities. For clearer visualization, the value ranges differ across benchmarks in each radar chart.
• Figure 2: Illustration of the Visual Jigsaw tasks. In the Image Jigsaw (top left), an image is partitioned into non-overlapping patches, shuffled into a sequence, and the model is tasked with predicting the correct raster order. In the Video Jigsaw (bottom), a video is segmented into temporal clips, shuffled, and the model predicts their original chronological order. In the 3D Jigsaw (top right), points with distinct depth values are sampled from an RGB-D image, shuffled and annotated in the RGB view, and the model is required to recover the correct depth order from nearest to farthest. Across all tasks, the policy model outputs an ordering that is compared against the ground truth, and a partial accuracy reward is assigned when only some elements are correctly ordered.
• Figure 3: Performance with different jigsaw difficulties on image and video tasks.
• Figure 4: Examples of the image jigsaw task. Each row shows a shuffled set of patches from an image, where the model is required to reconstruct the correct raster scan order. The ground-truth answers are displayed on the right.
• Figure 5: Example of the video jigsaw task. Each row shows a clip from the original video image, and the 6 clips are shuffled. The model is required to reconstruct the correct chronological order. The ground-truth answers are displayed on the right.