Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VALISENS: A Validated Innovative Multi-Sensor System for Cooperative Automated Driving

Lei Wan, Prabesh Gupta, Andreas Eich, Marcel Kettelgerdes, Hannan Ejaz Keen, Michael Klöppel-Gersdorf, Alexey Vinel

TL;DR

VALISENS addresses robust perception for automated driving by fusing onboard and roadside sensing across a cooperative multi-agent network using V2X communication. It employs LiDAR, radar, RGB, and thermal cameras with late, track-to-track fusion and a transformer-based trajectory predictor to create a unified environmental representation across vehicles and infrastructure. The system is validated in two urban test beds and a mobile rover, demonstrating improved VRU detection and extended perception range while incorporating sensor-condition monitoring for reliability. This work lays a practical foundation for Cooperative Intelligent Transport Systems by enabling scalable cooperative perception and communication across multi-agent networks.

Abstract

Perception is a core capability of automated vehicles and has been significantly advanced through modern sensor technologies and artificial intelligence. However, perception systems still face challenges in complex real-world scenarios. To improve robustness against various external factors, multi-sensor fusion techniques are essential, combining the strengths of different sensor modalities. With recent developments in Vehicle-to-Everything (V2X communication, sensor fusion can now extend beyond a single vehicle to a cooperative multi-agent system involving Connected Automated Vehicle (CAV) and intelligent infrastructure. This paper presents VALISENS, an innovative multi-sensor system distributed across multiple agents. It integrates onboard and roadside LiDARs, radars, thermal cameras, and RGB cameras to enhance situational awareness and support cooperative automated driving. The thermal camera adds critical redundancy for perceiving Vulnerable Road User (VRU), while fusion with roadside sensors mitigates visual occlusions and extends the perception range beyond the limits of individual vehicles. We introduce the corresponding perception module built on this sensor system, which includes object detection, tracking, motion forecasting, and high-level data fusion. The proposed system demonstrates the potential of cooperative perception in real-world test environments and lays the groundwork for future Cooperative Intelligent Transport Systems (C-ITS) applications.

VALISENS: A Validated Innovative Multi-Sensor System for Cooperative Automated Driving

TL;DR

VALISENS addresses robust perception for automated driving by fusing onboard and roadside sensing across a cooperative multi-agent network using V2X communication. It employs LiDAR, radar, RGB, and thermal cameras with late, track-to-track fusion and a transformer-based trajectory predictor to create a unified environmental representation across vehicles and infrastructure. The system is validated in two urban test beds and a mobile rover, demonstrating improved VRU detection and extended perception range while incorporating sensor-condition monitoring for reliability. This work lays a practical foundation for Cooperative Intelligent Transport Systems by enabling scalable cooperative perception and communication across multi-agent networks.

Abstract

Perception is a core capability of automated vehicles and has been significantly advanced through modern sensor technologies and artificial intelligence. However, perception systems still face challenges in complex real-world scenarios. To improve robustness against various external factors, multi-sensor fusion techniques are essential, combining the strengths of different sensor modalities. With recent developments in Vehicle-to-Everything (V2X communication, sensor fusion can now extend beyond a single vehicle to a cooperative multi-agent system involving Connected Automated Vehicle (CAV) and intelligent infrastructure. This paper presents VALISENS, an innovative multi-sensor system distributed across multiple agents. It integrates onboard and roadside LiDARs, radars, thermal cameras, and RGB cameras to enhance situational awareness and support cooperative automated driving. The thermal camera adds critical redundancy for perceiving Vulnerable Road User (VRU), while fusion with roadside sensors mitigates visual occlusions and extends the perception range beyond the limits of individual vehicles. We introduce the corresponding perception module built on this sensor system, which includes object detection, tracking, motion forecasting, and high-level data fusion. The proposed system demonstrates the potential of cooperative perception in real-world test environments and lays the groundwork for future Cooperative Intelligent Transport Systems (C-ITS) applications.
Paper Structure (21 sections, 6 figures, 3 tables)

This paper contains 21 sections, 6 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Communication
  4. Perception
  5. Sensor Condition Monitoring
  6. SYSTEM ARCHITECTURE
  7. Overview
  8. Infrastructure systems
  9. Vehicle systems
  10. Environmental perception concept
  11. LiDAR
  12. Radar
  13. RGB-T
  14. Intra-Entity Fusion
  15. Communication and Inter-Entity Fusion
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Sensor layout at the SmartIntersection in Dresden. The red circle shows the detection range of the LiDAR, the yellow ellipses show the detection area of the single radar sensors, and the blue (thermal) and green (thermal+RGB) polygons show the area covered by cameras.
  • Figure 2: Sensor configurations of the mobile sensor platform
  • Figure 3: Sensor configurations on test vehicles used in the experiments.
  • Figure 4: Software architecture overview of environmental perception.
  • Figure 5: Schematic diagrams of intra-entity and inter-entity fusion approaches
  • ...and 1 more figures