Evaluation Framework for Sensor Configuration Impact on Deep Learning-Based Perception
A Gamage, V Donzella
TL;DR
This paper addresses the challenge of evaluating DL-based perception under real-world onboard sensor inputs by introducing a modular, simulation‑driven framework that varies perception sensor modalities and parameters within operational design domains. The methodology couples synthetic data generation, preprocessing, inference feeding, and performance evaluation to quantify how sensor configuration impacts DL perception outputs, using RMSE as the evaluation metric. A case study on surround-vehicle trajectory prediction demonstrates that radar data with a narrow HFOV can yield predictions closer to ground truth than camera data, highlighting the nuanced tradeoffs between modalities. The framework offers a practical, scalable tool for OEMs and regulators to optimize sensor suites and establish robust, cost‑effective perception systems for automated driving.
Abstract
Current research on automotive perception systems predominantly focusses on either improving the performance of sensor technology or enhancing the perception functions in isolation. High-level perception functions are increasingly based on deep learning (DL) models due to their improved performance and generalisability compared to traditional algorithms. Despite the vital need to evaluate the performance of DL-based perception functions under real-world conditions using onboard sensor inputs, there is a lack of frameworks to implement such systematic evaluations. This paper presents a versatile framework to evaluate the impact of perception sensor modalities and parameter settings on DL-based perception functions. Using a simulation environment, the framework facilitates sensor modality selection and parameter tuning under different operational design domain conditions. Its effectiveness is demonstrated through a case study involving a state-of-the-art surround trajectory prediction model, highlighting performance differences across the sensor modalities radar and camera. Different settings for the parameter, horizontal field of view (HFOV) were evaluated to identify the optimal configuration. The results indicate that a radar sensor with a narrow HFOV is the most suitable configuration for the evaluated perception algorithm. The proposed framework offers a holistic approach to the design of the perception sensor suite, significantly contributing to the development of robust perception systems for automated driving systems.