Implementation Analysis of Collaborative Robot Digital Twins in Physics Engines

TL;DR

This paper tackles the challenge of implementing a digital twin for collaborative robotics by building a bidirectionally connected DT of two Franka cobots controlled via ROS 2 and synchronized to a high-precision infrared mocap system within Unreal Engine 5. The authors detail engineering choices, including engine selection, robot modeling pipelines (URDF-to-.gltf via Phobos/Blender), and a custom UDP/TCP networking scheme achieving 60 Hz dt updates and robust handling of tracking occlusions. They also introduce safety and analytics features such as Prohibited Zones and Log/Replay, and analyze network latency, jitter mitigation, and presentation options like Pixel Streaming. The work provides a practical, replicable framework that can guide DT deployment in robotics and automation research, with suggested future work in XR integration, scheduling optimizations, and multimodel AI integrations.

Abstract

This paper presents a Digital Twin (DT) of a 6G communications system testbed that integrates two robotic manipulators with a high-precision optical infrared tracking system in Unreal Engine 5. Practical details of the setup and implementation insights provide valuable guidance for users aiming to replicate such systems, an endeavor that is crucial to advancing DT applications within the scientific community. Key topics discussed include video streaming, integration within the Robot Operating System 2 (ROS 2), and bidirectional communication. The insights provided are intended to support the development and deployment of DTs in robotics and automation research.

Figures (4)

• Figure 1: Image of the teleoperation test environment for communication systems, with its dt visualized in the background. The cobot on the left represents the leading system (Leader), while the cobot on the right represents the following system (Follower).
• Figure 2: Abstraction of data exchange in the setup. The bidirectional communication based on ros2 of the teleoperation setup is indicated by the violet arrow. Communication between the cobots and their respective dt is established via UDP and TCP connections. The joint sensor data is extracted from respective ros2 messages and transmitted via UDP connections (orange arrow), while command data from the dt is transmitted via TCP connections (blue arrow) and converted into ros2 messages. Along with coordinates of the cobots provided by the precision positioning system (green dashed arrow), all data is merged in the dt of the respective cobot.
• Figure 3: A screenshot of the dt environment displays the system representation, with the leader station on the left and the follower station on the right. In the rear corners of the stations, the individual marker objects are depicted. At the front left of the follower station, a vase is visible, accompanied with a bounding box delineating a prohibited area. In the background, the visualized scene from a virtual camera is presented.
• Figure 4: Time differences in processing steps from Wireshark to digital twin. n=200k. Delta times from Wireshark Capture to receive time in unreal (left). The processing time needed in unreal from receive time and package applied (middle). Combined times (right).