Activating Visual Context and Commonsense Reasoning through Masked Prediction in VLMs

TL;DR

This work addresses the gap in robust multimodal reasoning by introducing Masked Prediction via Context and Commonsense (MPCC), a task that forces vision-language models to fuse visual context with commonsense knowledge to reconstruct occluded content. The authors build MPCC-Eval, a COCO-based benchmark with varying difficulty and confounding items, and explore reasoning-guided fine-tuning strategies including prompt-based guiding, supervised fine-tuning, and reinforcement fine-tuning using GRPO, complemented by a prior sampling approach to mix annotated and model-generated reasoning. Results show that reinforcement fine-tuning with prior sampling offers strong generalization, especially under distribution shifts, and that MPCC fine-tuning improves cross-task performance on downstream reasoning tasks, albeit with some trade-offs in in-distribution accuracy. Overall, MPCC provides a low-cost, scalable path to enhance visual context activation and commonsense reasoning in VLMs, with MPCC-Eval enabling robust assessment and guiding future developments in multimodal reasoning.

Abstract

Recent breakthroughs in reasoning models have markedly advanced the reasoning capabilities of large language models, particularly via training on tasks with verifiable rewards. Yet, a significant gap persists in their adaptation to real world multimodal scenarios, most notably, vision language tasks, due to a heavy focus on single modal language settings. While efforts to transplant reinforcement learning techniques from NLP to VLMs have emerged, these approaches often remain confined to perception centric tasks or reduce images to textual summaries, failing to fully exploit visual context and commonsense knowledge, ultimately constraining the generalization of reasoning capabilities across diverse multimodal environments. To address this limitation, we introduce a novel fine tuning task, Masked Prediction via Context and Commonsense, which forces models to integrate visual context and commonsense reasoning by reconstructing semantically meaningful content from occluded images, thereby laying the foundation for generalized reasoning. To systematically evaluate the model performance in generalized reasoning, we developed a specialized evaluation benchmark, MPCC Eval, and employed various fine tuning strategies to guide reasoning. Among these, we introduced an innovative training method, Reinforcement Fine tuning with Prior Sampling, which not only enhances model performance but also improves its generalized reasoning capabilities in OOD and cross task scenarios.

Figures (4)

• Figure 1: The example of Masked Prediction via Context and Commonsense (MPCC) task. In this example, we highlighted the visual context reasoning part in blue and the commonsense reasoning part in orange.
• Figure 2: The MPCC-Eval benchmark creation pipeline filters and curates images by difficulty, forming three subsets: easy, moderate, and hard. It includes two types of single-choice question formats.
• Figure 3: Fine-tuning Strategies on MPCC Task. (a) SFT: Construct and distill reasoning data via supervised fine-tuning. (b) GRPO-based RFT: Activate reasoning with verifiable rewards. (c) Proposed RFT with Prior Sampling: Use annotated reasoning data to replace one sampling step.
• Figure 4: Model responses in the MPCC task and high-frequency word statistics after prompt engineering and fine-tuning.