PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations

TL;DR

PRISM addresses the problem of learning robust robotic policies from few demonstrations by constructing scene-consistent simulation environments from a single scene image and a 3D model library. It introduces a projection-based reward model supervised by vision-language models using human-guided object projection relationships, and performs co-training with expert demonstrations to bridge the sim-to-real gap. The framework combines real-to-sim environment generation, multi-view reward labeling, and an action feasibility predictor to achieve robust policy transfer, outperforming imitation-learning baselines and prior VLM-based approaches across six manipulation tasks. This approach enables reliable real-world deployment with limited real data, reducing the need for handcrafted rewards and extensive real-world trials.

Abstract

Learning from few demonstrations to develop policies robust to variations in robot initial positions and object poses is a problem of significant practical interest in robotics. Compared to imitation learning, which often struggles to generalize from limited samples, reinforcement learning (RL) can autonomously explore to obtain robust behaviors. Training RL agents through direct interaction with the real world is often impractical and unsafe, while building simulation environments requires extensive manual effort, such as designing scenes and crafting task-specific reward functions. To address these challenges, we propose an integrated real-to-sim-to-real pipeline that constructs simulation environments based on expert demonstrations by identifying scene objects from images and retrieving their corresponding 3D models from existing libraries. We introduce a projection-based reward model for RL policy training that is supervised by a vision-language model (VLM) using human-guided object projection relationships as prompts, with the policy further fine-tuned using expert demonstrations. In general, our work focuses on the construction of simulation environments and RL-based policy training, ultimately enabling the deployment of reliable robotic control policies in real-world scenarios.

Figures (6)

• Figure 1: Human-guided Object Projection Relationships. Multi-view simulated images are generated from scene view, and human-guided object projection relationships are used as prompts to query the VLM to evaluate whether each view satisfies the task-specific spatial requirement.
• Figure 2: PRISM System Overview. 1) Transfer the real-world scene to the simulator by estimating object poses in the environment and collecting simulation data (see Section \ref{['sec:Real-to-Sim Transfer for Scene-Aware Simulation Environment Generation']}). 2) Train the reward model using human-guided object projection relationships and apply it to RL. Injecting initialization noise in the simulation enhances the robustness of the control policy (see Section \ref{['sec:Robustifying Real-World Imitation Learning Policies in Simulation']}). 3) Fine-tune the learned policy from simulation using few real-world demonstrations and train an action feasibility predictor model (see Section \ref{['sec: Co-Training on Real-World Data for Sim-to-Real Transfer']}). 4) Evaluate policy stability in real-world tasks, ensuring robust behaviors that generalize to novel robot initial states and object poses.
• Figure 3: Two-stage VLM querying process for task-specific reward labeling. The pre-task prompt evaluates whether the current state satisfies the condition for executing the action, while the post-task prompt evaluates task completion. This template generalizes across all similar tasks.
• Figure 4: Qualitative Results for Real-world Robot Experiments. The simulation environment is constructed based on RGB-D images and a 3D model library, with additional noise injected to facilitate the learning of a robust control policy. The policy is evaluated on six tasks in the real world. The green dashed lines denote the approximate placement regions for object randomization.
• Figure 5: Results for Action Feasibility Predictor. The first row shows positions that typically lead to task failure, while the second row displays positions suitable for successful execution.