Demonstrating DVS: Dynamic Virtual-Real Simulation Platform for Mobile Robotic Tasks

TL;DR

This work presents Dynamic Virtual-Real Simulation (DVS), a platform designed to close the sim-to-real gap in mobile robotics by fusing high-fidelity virtual environments with real-world feedback. It combines virtual-real fusion, dynamic scene generation including pedestrian and multi-robot plugins, optical motion capture precision, and ROS-based bidirectional communication to enable closed-loop training and validation. An intervention-enabled workflow allows real-time scenario adjustments during physical execution, and experiments demonstrate improvements in tasks such as grasping, trajectory prediction, and social navigation. The approach offers a practical pathway toward robust real-world deployment and enhanced human-robot collaboration in dynamic indoor environments, with explicit mechanisms for precise pose synchronization and real-time feedback.

Abstract

With the development of embodied artificial intelligence, robotic research has increasingly focused on complex tasks. Existing simulation platforms, however, are often limited to idealized environments, simple task scenarios and lack data interoperability. This restricts task decomposition and multi-task learning. Additionally, current simulation platforms face challenges in dynamic pedestrian modeling, scene editability, and synchronization between virtual and real assets. These limitations hinder real world robot deployment and feedback. To address these challenges, we propose DVS (Dynamic Virtual-Real Simulation Platform), a platform for dynamic virtual-real synchronization in mobile robotic tasks. DVS integrates a random pedestrian behavior modeling plugin and large-scale, customizable indoor scenes for generating annotated training datasets. It features an optical motion capture system, synchronizing object poses and coordinates between virtual and real world to support dynamic task benchmarking. Experimental validation shows that DVS supports tasks such as pedestrian trajectory prediction, robot path planning, and robotic arm grasping, with potential for both simulation and real world deployment. In this way, DVS represents more than just a versatile robotic platform; it paves the way for research in human intervention in robot execution tasks and real-time feedback algorithms in virtual-real fusion environments. More information about the simulation platform is available on https://immvlab.github.io/DVS/.

Figures (5)

• Figure 1: Overview of DVS platform, which offers a variety of large-scale indoor scene types and dynamic element plugins on the left, enabling users to construct dynamic environments. In the middle, the platform supports various data types that can be generated, such as RGB, depth, and semantic labels. On the right, the data created using this platform can be applied to train robots for tasks such as navigation, trajectory prediction, and grasping. Through a virtual-real fusion feedback mechanism, the platform allows bidirectional mapping of the states of real and virtual agents, enriching the research scenarios.
• Figure 2: Virtual-Real Data Synchronization Framework. The central demonstrates the synchronization of object pose and robot motion through VRPN and ROS. The left and right parts depict the virtual simulation environment and physical real world scene, respectively.
• Figure 3: The interactive interface of the simulation platform: The left panel adjusts dynamic pedestrian parameters while the right selects perception data types.
• Figure 4: The robotic arm is interrupted while executing Prompt A and is requested to execute Prompt B. The first row shows the robotic arm in the virtual platform, and the second row shows the real robotic arm.
• Figure 5: Visualization of pedestrian trajectory prediction, where each color represents a different pedestrian. The accuracy of the prediction is higher when the predicted trajectory (short dashed line) closely aligns with the ground truth (GT, solid line). In environments with dense static obstacles, such as indoors, the predicted future trajectory may result in collisions (red rectangular box).