Compressing Sensor Data for Remote Assistance of Autonomous Vehicles using Deep Generative Models

TL;DR

The paper tackles the challenge of transmitting sensor data for remote assistance in autonomous vehicles under real-time constraints by evaluating deep generative models for image and lidar compression. It presents offline evaluations of a VAE with hyperprior and a GAN for images and a VAE-based approach for lidar, plus an online ROS/CARLA pipeline trained on KITTI data. Results show GANs deliver superior reconstruction quality at low bit-rates but face latency hurdles, whereas VAEs offer a more practical trade-off for real-time remote assistance; lidar results are promising but limited by preprocessing and system overhead. The work informs practical design choices for remote-assistance systems and points to future improvements in cross-modality processing, latency optimization, and deployment on larger datasets.

Abstract

In the foreseeable future, autonomous vehicles will require human assistance in situations they can not resolve on their own. In such scenarios, remote assistance from a human can provide the required input for the vehicle to continue its operation. Typical sensors used in autonomous vehicles include camera and lidar sensors. Due to the massive volume of sensor data that must be sent in real-time, highly efficient data compression is elementary to prevent an overload of network infrastructure. Sensor data compression using deep generative neural networks has been shown to outperform traditional compression approaches for both image and lidar data, regarding compression rate as well as reconstruction quality. However, there is a lack of research about the performance of generative-neural-network-based compression algorithms for remote assistance. In order to gain insights into the feasibility of deep generative models for usage in remote assistance, we evaluate state-of-the-art algorithms regarding their applicability and identify potential weaknesses. Further, we implement an online pipeline for processing sensor data and demonstrate its performance for remote assistance using the CARLA simulator.

Figures (11)

• Figure 1: Overview of the online processing pipeline. The pre-processed lidar data passes through the same encoder and decoder architecture as RGB data.
• Figure 2: Comparison of VAE, GAN and JPEG2000 compression with the metrics MS-SSIM, LPIPS, MSE and PSNR. The performances are plotted over the bit-rate in bits per pixel.
• Figure 3: KITTI street scene with reconstruction comparisons. Target bit-rate is approx. $0.8\,bpp$.
• Figure 4: Relative error of the number of detected cars in the original image and the reconstructed image by VAE respectively GAN. The distribution is given over the bit-rate in bpp. Scott's rule kde_scott was used for smoothing with a scaling factor of 0.6. The darker the color in the graph, the higher the density of the values.
• Figure 5: Object detection example performed in the original image and several reconstructions made by GAN and VAE with differing bit-rates.