RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception

TL;DR

RLS3 tackles limited and imbalanced fine-tuning data for vision–language models by introducing an RL-driven synthetic data generator that actively crafts informative indoor scenes. The RL agent manipulates objects in a Unity simulation and receives feedback from the VLM to generate targeted, spatial-relations data, which is used to fine-tune the VLM in a loop guided by an extrinsic VLM-based reward. The approach demonstrates data-efficient improvements in spatial reasoning for both CLIP-type models and PaliGemma, with careful prompt generation and a dual-loss VLM objective. The framework has potential to generalize to other multimodal tasks and scale with larger models, while highlighting directions for sim2real transfer and finer-grained reinforcement signals.

Abstract

Vision-language model (VLM) fine-tuning for application-specific visual grounding based on natural language instructions has become one of the most popular approaches for learning-enabled autonomous systems. However, such fine-tuning relies heavily on high-quality datasets to achieve successful performance in various downstream tasks. Additionally, VLMs often encounter limitations due to insufficient and imbalanced fine-tuning data. To address these issues, we propose a new generalizable framework to improve VLM fine-tuning by integrating it with a reinforcement learning (RL) agent. Our method utilizes the RL agent to manipulate objects within an indoor setting to create synthetic data for fine-tuning to address certain vulnerabilities of the VLM. Specifically, we use the performance of the VLM to provide feedback to the RL agent to generate informative data that efficiently fine-tune the VLM over the targeted task (e.g. spatial reasoning). The key contribution of this work is developing a framework where the RL agent serves as an informative data sampling tool and assists the VLM in order to enhance performance and address task-specific vulnerabilities. By targeting the data sampling process to address the weaknesses of the VLM, we can effectively train a more context-aware model. In addition, generating synthetic data allows us to have precise control over each scene and generate granular ground truth captions. Our results show that the proposed data generation approach improves the spatial reasoning performance of VLMs, which demonstrates the benefits of using RL-guided data generation in vision-language tasks.

Figures (8)

• Figure 1: A detailed overview of our proposed RLS3 framework. The scheduler acts as a synchronizer between the processes. The loop begins with data generation where the RL agent takes $T$ timesteps in which $T_0 \leq T$ image-metadata samples are generated from the Unity environment. In each step, the agent receives an intrinsic reward based on the feasibility of the generated sample. The episode is paused on the last step to allow the following processes to complete. Next, the metadata is used in the prompt generation process to create captions describing a spatial relation. The $T_0$ image-text pairs are then inputted to the VLM for inference. The performance of the VLM is used as an extrinsic reward signal $J_2$ for the RL agent at the end of the episode. Steps 1, 2, and 3 repeat for $E$ RL episodes to generate a diverse batch of data for fine-tuning. Diversity is further increased in the fine-tuning batch by sampling the generated data with sampling rate $\eta$ for each episode, resulting in a fine-tuning batch size of $E \eta T_0$. After fine-tuning the VLM finished, the batch is cleared and the process repeats for the next iteration.
• Figure 2: A diagram showing the Unity environment structure. There is one active scene at a time, which is cycled over the episode. Each step selects one of the three active objects and is swapped with another in the object container. Only a handful of the available objects are shown here.
• Figure 3: Illustration of our template-based prompt generation. The center of 'Object A' is located on the origin of the eight regions in the horizontal direction and 3 regions in the vertical direction in which spatial terms are selected. The horizontal regions are aligned such that 'behind' is facing towards the camera. The cameras are aligned with the horizontal and vertical axes in this figure to more clearly show the regions. The generated caption for this scenario is: "The small pot is above, behind and to the left of the yellow bowl."
• Figure 4: Combined figures of PaliGemma score and CLIP performance vs cumulative generated data.
• Figure 5: PaliGemma score ($avg \pm std$) separated by spatial term for RLS3 with an SAC and random agent. Cumulative term counts of data generated for fine-tuning are given in ( ) for SAC and [ ] for random agents.