Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CD-TWINSAFE: A ROS-enabled Digital Twin for Scene Understanding and Safety Emerging V2I Technology

Amro Khaled, Farah Khaled, Omar Riad, Catherine M. Elias

TL;DR

CD-TWINSAFE addresses real-time scene understanding and safety for autonomous vehicles by integrating an on-board perception/localization stack with an Unreal Engine 5–based digital twin that mirrors the ego vehicle and detected objects via a ROS2 UDP channel. The approach uses stereo-vision to run two perception pipelines for object detection/depth estimation and safety feature extraction, producing metrics such as TTC and THW, and delivering safety alerts to the cockpit or remote operator. The architecture enables V2I communication over 4G with a lightweight ROS2 DDS-based message format and real-time DT visualization and control capabilities. The work demonstrates real-time performance across driving scenarios, highlighting the digital twin's utility for monitoring, alerting, and potential remote maneuvering, while noting limitations like depth range and sensor drift with future fusion of LiDAR/radar.

Abstract

In this paper, the CD-TWINSAFE is introduced, a V2I-based digital twin for Autonomous Vehicles. The proposed architecture is composed of two stacks running simultaneously, an on-board driving stack that includes a stereo camera for scene understanding, and a digital twin stack that runs an Unreal Engine 5 replica of the scene viewed by the camera as well as returning safety alerts to the cockpit. The on-board stack is implemented on the vehicle side including 2 main autonomous modules; localization and perception. The position and orientation of the ego vehicle are obtained using on-board sensors. Furthermore, the perception module is responsible for processing 20-fps images from stereo camera and understands the scene through two complementary pipelines. The pipeline are working on object detection and feature extraction including object velocity, yaw and the safety metrics time-to-collision and time-headway. The collected data form the driving stack are sent to the infrastructure side through the ROS-enabled architecture in the form of custom ROS2 messages and sent over UDP links that ride a 4G modem for V2I communication. The environment is monitored via the digital twin through the shared messages which update the information of the spawned ego vehicle and detected objects based on the real-time localization and perception data. Several tests with different driving scenarios to confirm the validity and real-time response of the proposed architecture.

CD-TWINSAFE: A ROS-enabled Digital Twin for Scene Understanding and Safety Emerging V2I Technology

TL;DR

CD-TWINSAFE addresses real-time scene understanding and safety for autonomous vehicles by integrating an on-board perception/localization stack with an Unreal Engine 5–based digital twin that mirrors the ego vehicle and detected objects via a ROS2 UDP channel. The approach uses stereo-vision to run two perception pipelines for object detection/depth estimation and safety feature extraction, producing metrics such as TTC and THW, and delivering safety alerts to the cockpit or remote operator. The architecture enables V2I communication over 4G with a lightweight ROS2 DDS-based message format and real-time DT visualization and control capabilities. The work demonstrates real-time performance across driving scenarios, highlighting the digital twin's utility for monitoring, alerting, and potential remote maneuvering, while noting limitations like depth range and sensor drift with future fusion of LiDAR/radar.

Abstract

In this paper, the CD-TWINSAFE is introduced, a V2I-based digital twin for Autonomous Vehicles. The proposed architecture is composed of two stacks running simultaneously, an on-board driving stack that includes a stereo camera for scene understanding, and a digital twin stack that runs an Unreal Engine 5 replica of the scene viewed by the camera as well as returning safety alerts to the cockpit. The on-board stack is implemented on the vehicle side including 2 main autonomous modules; localization and perception. The position and orientation of the ego vehicle are obtained using on-board sensors. Furthermore, the perception module is responsible for processing 20-fps images from stereo camera and understands the scene through two complementary pipelines. The pipeline are working on object detection and feature extraction including object velocity, yaw and the safety metrics time-to-collision and time-headway. The collected data form the driving stack are sent to the infrastructure side through the ROS-enabled architecture in the form of custom ROS2 messages and sent over UDP links that ride a 4G modem for V2I communication. The environment is monitored via the digital twin through the shared messages which update the information of the spawned ego vehicle and detected objects based on the real-time localization and perception data. Several tests with different driving scenarios to confirm the validity and real-time response of the proposed architecture.
Paper Structure (19 sections, 8 equations, 7 figures, 2 tables)

This paper contains 19 sections, 8 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Studies
  3. Methodology
  4. Perception and Localization Modules
  5. The Localization Module
  6. The Vision-based Perception Module
  7. Stage 1: Object Detection and Classification
  8. Stage 2: Depth Estimation
  9. Stage 5: Safety Features Extraction
  10. On-board User Interface
  11. Communication
  12. On-Board Communication
  13. V2I Protocols
  14. Infrastructure - Digital Twin
  15. Class Structure
  16. ...and 4 more sections

Figures (7)

  • Figure 1: The CD-TWINSAFE Architecture Overview
  • Figure 2: On-board UI showing the state as background color and messages from digital twin operator.
  • Figure 3: (a) Message Format Used for Byte Encoding, (b) Implemented Class Structure of Digital Twin
  • Figure 4: Snapshot of Digital Twin UI
  • Figure 5: Values of perception pipeline 1 and 2 in comparison
  • ...and 2 more figures