SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning

TL;DR

SSL4RL presents a unified framework that repurposes self-supervised learning tasks as verifiable reinforcement learning rewards for post-training vision-language models. By using a corruption-based environment and GRPO optimization, it achieves consistent gains across vision-centric and multimodal reasoning benchmarks, without relying on human labels or external verifiers. The study also reveals design principles, such as task-domain alignment, difficulty matching, and data-scale considerations, and demonstrates cross-domain applicability to graph data. Overall, SSL4RL offers a scalable path to improve grounding and reasoning in multimodal models while highlighting the need for adaptive task design for future scale.

Abstract

Vision-language models (VLMs) have shown remarkable abilities by integrating large language models with visual inputs. However, they often fail to utilize visual evidence adequately, either depending on linguistic priors in vision-centric tasks or resorting to textual shortcuts during reasoning. Although reinforcement learning (RL) can align models with desired behaviors, its application to VLMs has been hindered by the lack of scalable and reliable reward mechanisms. To overcome this challenge, we propose SSL4RL, a novel framework that leverages self-supervised learning (SSL) tasks as a source of verifiable rewards for RL-based fine-tuning. Our approach reformulates SSL objectives-such as predicting image rotation or reconstructing masked patches-into dense, automatic reward signals, eliminating the need for human preference data or unreliable AI evaluators. Experiments show that SSL4RL substantially improves performance on both vision-centric and vision-language reasoning benchmarks. Furthermore, through systematic ablations, we identify key factors-such as task difficulty, model scale, and semantic alignment with the target domain-that influence the effectiveness of SSL4RL tasks, offering new design principles for future work. We also demonstrate the framework's generality by applying it to graph learning, where it yields significant gains. SSL4RL establishes a versatile and effective paradigm for aligning multimodal models using verifiable, self-supervised objectives.

Figures (13)

• Figure 1: Overview of the SSL4RL framework. A corruption function transforms an input into a context--target pair. The model conditions on the context, generates predictions, and receives a verifiable reward by comparing against the target. The reward is then used to optimize the model via Reinforcement Learning (RL).
• Figure 2: Four SSL4RL tasks considered in our study. Rotation: An image is rotated by a predefined angle, and the task is to predict this angle. Jigsaw: After dividing an image into a grid and permuting the patches, the goal is to predict the correct permutation index. Contrastive: Two augmented views are generated from an image, and the objective is to identify whether two given views originate from the same source image. Position: Given an image and a single patch cropped from it, the task is to predict the patch's original spatial position.
• Figure 3: Cross-Attention Heatmap Comparison. More instances are shown in Appendix \ref{['appen:case_analysis']}.
• Figure 4: Test accuracy (%) on MMBench, varying the SSL task difficulty.
• Figure 5: Test accuracy of SSL tasks.