SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments

TL;DR

SimPRIVE tackles the challenge of safely testing AI-driven controllers for physical robots in cyber-physical systems by linking a real robot to a photo-realistic virtual world rendered in Unreal Engine 5 via ROS 2. It implements a vehicle-in-the-loop framework with a digital twin that mirrors real robot motion and generates synthetic sensor data for testing reinforcement learning and perception algorithms, reducing risk and cost. The paper details the architecture, key callbacks (pose, reset, pause/resume), and optional modes, and validates the approach with an obstacle-avoidance RL agent on an AgileX Scout Mini navigating a virtual corridor using LiDAR and camera data. Results show real-time performance with mostly sub-100 ms latency and effective safety mechanisms, highlighting SimPRIVE's potential for rapid, safe experimentation in robotics.

Abstract

The use of machine learning in cyber-physical systems has attracted the interest of both industry and academia. However, no general solution has yet been found against the unpredictable behavior of neural networks and reinforcement learning agents. Nevertheless, the improvements of photo-realistic simulators have paved the way towards extensive testing of complex algorithms in different virtual scenarios, which would be expensive and dangerous to implement in the real world. This paper presents SimPRIVE, a simulation framework for physical robot interaction with virtual environments, which operates as a vehicle-in-the-loop platform, rendering a virtual world while operating the vehicle in the real world. Using SimPRIVE, any physical mobile robot running on ROS 2 can easily be configured to move its digital twin in a virtual world built with the Unreal Engine 5 graphic engine, which can be populated with objects, people, or other vehicles with programmable behavior. SimPRIVE has been designed to accommodate custom or pre-built virtual worlds while being light-weight to contain execution times and allow fast rendering. Its main advantage lies in the possibility of testing complex algorithms on the full software and hardware stack while minimizing the risks and costs of a test campaign. The framework has been validated by testing a reinforcement learning agent trained for obstacle avoidance on an AgileX Scout Mini rover that navigates a virtual office environment where everyday objects and people are placed as obstacles. The physical rover moves with no collision in an indoor limited space, thanks to a LiDAR-based heuristic.

Figures (9)

• Figure 1: High-level overview of the architecture of the proposed SimPRIVE framework. The figure also shows renderings of the digital twin of the rover in different virtual environments, with synthetic LiDAR data superimposed.
• Figure 2: Definition of the Digital, Physical and World frames and corresponding transformations. The pose of the digital frame ($T_{W,k}^D$) is computed from the physical one ($T_{W,k}^P$) by transforming it with the offset $T_{P,0}^D$. The timeline on the bottom of the image illustrates the working principles of the Pause/Resume callbacks and how the offset is updated.
• Figure 3: Functional control architecture of the rover on-board software used for the experiments. The arrows specify the dedicated SimPRIVE ROS 2 topics.
• Figure 4: Path of the physical rover (top) and corresponding path of its digital twin (bottom). Each trajectory changes color from red (at the starting point) to green (at the final point) to better compare the positions in the physical and digital world.
• Figure 5: Distribution of the round-trip latency times (top) and the UE5 execution times for 10,000 iterations in log scale (bottom). While the rendering in UE5 shows predictable execution time, ROS2 occasionally introduces spurious delays. However, 99.82% of the total measures result to be less than 100 ms.