Simulating Automotive Radar with Lidar and Camera Inputs

TL;DR

The paper tackles the lack of quality automotive radar datasets by proposing a data-driven approach to simulate $4D$ mmWave radar signals and RSS from camera images, LiDAR, and ego-velocity. It introduces Dis-Net, which predicts radar signal distribution and count, and RSS-Net, which estimates RSS by fusing appearance and local geometry, together enabling end-to-end generation of synthetic radar datagrams. The method is validated on three open radar datasets across different models, achieving high-fidelity radar signal synthesis and improved object-detection performance when using synthesized radar data for training. This work enables radar-aware development and evaluation using readily available vision data, advancing simulators and enabling more robust radar-based autonomous navigation without relying on specialized radar corpora.

Abstract

Low-cost millimeter automotive radar has received more and more attention due to its ability to handle adverse weather and lighting conditions in autonomous driving. However, the lack of quality datasets hinders research and development. We report a new method that is able to simulate 4D millimeter wave radar signals including pitch, yaw, range, and Doppler velocity along with radar signal strength (RSS) using camera image, light detection and ranging (lidar) point cloud, and ego-velocity. The method is based on two new neural networks: 1) DIS-Net, which estimates the spatial distribution and number of radar signals, and 2) RSS-Net, which predicts the RSS of the signal based on appearance and geometric information. We have implemented and tested our method using open datasets from 3 different models of commercial automotive radar. The experimental results show that our method can successfully generate high-fidelity radar signals. Moreover, we have trained a popular object detection neural network with data augmented by our synthesized radar. The network outperforms the counterpart trained only on raw radar data, a promising result to facilitate future radar-based research and development.

Figures (8)

• Figure 1: Properties of mmWave radar point cloud and the associated challenges of data synthesis: (a) Signal inconsistency: two radar point clouds (brown and blue) from the same perspective do not have the same point positions. (b) Large depth variations: Green points are where radar points agrees lidar points in depth, red points otherwise. (c) Large point dispersion: radar signals reflected by a certain object may "float" around it. (d) Cross-modality sensitivity difference: radar (red) and lidar (blue) point cloud possess different reflection sensitivity.
• Figure 2: Coordinate systems illustration.
• Figure 3: Algorithm pipeline for generating simulated radar signals (inference stage). The algorithm takes lidar point cloud, camera image, and radar ego-velocty as input. The grayscale image represents the predicted radar signal distribution with high insensitivity representing high probability. The size of the red dot in the lower right image indicates the predicted RSS.
• Figure 4: Overview of Dis-Net. The first branch generates a grayscale image as the probability density function of radar signals' distribution, and the second branch fits the number of signals. Avg Pool refers to average pooling, Adp Pool refers to adaptive pooling, Conv 2D refers to 2D convolutional layer, FC refers to fully connected layer (dense layer), TransposeConv 2D refers to 2D transpose convolutional layer, and Concat refers to concatenation.
• Figure 5: Left: radar signal (red dots) projection on image, and Right: the corresponding signal distribution function illustrated in the grayscale image (translucent).

Theorems & Definitions (1)

• Definition 1