Towards Sensor Data Abstraction of Autonomous Vehicle Perception Systems

TL;DR

The paper tackles the problem that autonomous-vehicle perception models are biased by their sensor configurations, hindering transfer to new hardware. It proposes sensor data abstraction as a unified, learned interface that maps data from $1$-$n$ sensors to a common representation suitable for neural perception pipelines. It provides a taxonomy distinguishing marginal and joint abstraction, discusses leveraging metadata (intrinsics/extrinsics), and reviews practical pipelines across camera, lidar, and radar modalities. By outlining research directions and advocating simulation-based data support, the work aims to enable transferability and broader applicability of HAD perception across diverse sensor setups.

Abstract

Full-stack autonomous driving perception modules usually consist of data-driven models based on multiple sensor modalities. However, these models might be biased to the sensor setup used for data acquisition. This bias can seriously impair the perception models' transferability to new sensor setups, which continuously occur due to the market's competitive nature. We envision sensor data abstraction as an interface between sensor data and machine learning applications for highly automated vehicles (HAD). For this purpose, we review the primary sensor modalities, camera, lidar, and radar, published in autonomous-driving related datasets, examine single sensor abstraction and abstraction of sensor setups, and identify critical paths towards an abstraction of sensor data from multiple perception configurations.

Figures (3)

• Figure 1: Sensor data abstraction provides an interface between a sensor and further applications.
• Figure 2: Quantitative comparison between the three major sensor modalities. From bad (center) to excellent (rim).
• Figure 3: Data flow from sensors to an environment model. The abstraction layer allows for a decoupling of observations and the state of a system.