RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding

TL;DR

RGBX-R1 addresses RGB-centric limitations in multimodal large language models by extending perception and reasoning to infrared, depth, and event modalities. It introduces Visual Modality Chain-of-Thought (VM-CoT) prompts via UAV-based Understand–Associate–Validate guidance, and a two-stage training pipeline consisting of Cold-Start Supervised Fine-Tuning (CS-SFT) with Modality-specific Token Weighting (MTW) and Spatio-Temporal Reinforcement Fine-Tuning (ST-RFT) with the MuST reward. A new RGBX-Grounding benchmark and dataset enable evaluation of cross-modal grounding, where RGBX-R1 achieves substantial gains (e.g., 22.71% over baselines) and exhibits Modal Knowledge Emergent, transferring RGB understanding to unseen X modalities with limited data. The work demonstrates improved robustness in grounding under degraded conditions and provides a framework for scalable cross-modal reasoning in multimodal grounding tasks.

Abstract

Multimodal Large Language Models (MLLM) are primarily pre-trained on the RGB modality, thereby limiting their performance on other modalities, such as infrared, depth, and event data, which are crucial for complex scenarios. To address this, we propose RGBX-R1, a framework to enhance MLLM's perception and reasoning capacities across various X visual modalities. Specifically, we employ an Understand-Associate-Validate (UAV) prompting strategy to construct the Visual Modality Chain-of-Thought (VM-CoT), which aims to expand the MLLMs' RGB understanding capability into X modalities. To progressively enhance reasoning capabilities, we introduce a two-stage training paradigm: Cold-Start Supervised Fine-Tuning (CS-SFT) and Spatio-Temporal Reinforcement Fine-Tuning (ST-RFT). CS-SFT supervises the reasoning process with the guidance of VM-CoT, equipping the MLLM with fundamental modality cognition. Building upon GRPO, ST-RFT employs a Modality-understanding Spatio-Temporal (MuST) reward to reinforce modality reasoning. Notably, we construct the first RGBX-Grounding benchmark, and extensive experiments verify our superiority in multimodal understanding and spatial perception, outperforming baselines by 22.71% on three RGBX grounding tasks.

Figures (13)

• Figure 1: Existing MLLMs exhibit limited capabilities with non-RGB visual modalities and struggle to leverage modality complementarity to enhance multimodal grounding.
• Figure 2: Overall framework of RGBX-R1. (a) RGBX MIG data construction. (b) Constructing VM-CoT with the UAV prompting strategy and two-stage filtering. (c) Cold-start Supervised Fine-Tuning equipped with the MTW mechanism, this stage aims to guide the model in performing structured reasoning. (d) Spatio-Temporal Reinforcement Fine-Tuning. The policy model generates multiple responses and optimizes the model via the MuST group rewards.
• Figure 3: The curves of rewards during ST-RFT.
• Figure 4: Grounding performance of RGBX-R1-7B under different modality inputs to illustrate modality contributions.
• Figure 5: Visualization of RGBX grounding results. The green rectangles indicate targets in the template images.