Driving Policy Transfer via Modularity and Abstraction

TL;DR

The paper tackles the sim-to-real transfer problem for autonomous driving by introducing a modular architecture that separates perception, high-level policy, and low-level control. Perception produces a semantic road segmentation, while the driving policy outputs local waypoints from this segmentation, and a PID-based low-level controller executes the path. Trained entirely in simulation (with noisy segmentation from Cityscapes-inspired data), the policy transfers directly to a real 1/5-scale truck without finetuning, outperforming end-to-end baselines in both simulated and real-world tests. The results demonstrate that modularity and abstraction facilitate robust transfer and allow leveraging abundant simulation data while mitigating perception and dynamics gaps, marking a practical step toward real-world deployment of learned driving policies.

Abstract

End-to-end approaches to autonomous driving have high sample complexity and are difficult to scale to realistic urban driving. Simulation can help end-to-end driving systems by providing a cheap, safe, and diverse training environment. Yet training driving policies in simulation brings up the problem of transferring such policies to the real world. We present an approach to transferring driving policies from simulation to reality via modularity and abstraction. Our approach is inspired by classic driving systems and aims to combine the benefits of modular architectures and end-to-end deep learning approaches. The key idea is to encapsulate the driving policy such that it is not directly exposed to raw perceptual input or low-level vehicle dynamics. We evaluate the presented approach in simulated urban environments and in the real world. In particular, we transfer a driving policy trained in simulation to a 1/5-scale robotic truck that is deployed in a variety of conditions, with no finetuning, on two continents. The supplementary video can be viewed at https://youtu.be/BrMDJqI6H5U

Figures (12)

• Figure 1: System architecture. The autonomous driving system comprises three modules: a perception module implemented by an encoder-decoder network, a command-conditional driving policy implemented by a branched convolutional network, and a low-level PID controller.
• Figure 2: Waypoints are encoded by the distance to the vehicle and the relative angle to the vehicle's heading.
• Figure 3: Simulation environment. Maps of the two towns, along with example images that show the towns in two conditions: clear daytime (Weather 1) and cloudy daytime after rain (Weather 2). We use Town 1/Weather 1 during training. The other three combinations (Town 1/Weather 2, Town 2/Weather 1, and Town 2/Weather 2) are used to evaluate generalization in simulation. Note the significant visual differences between the towns and weather conditions.
• Figure 4: Quantitative evaluation of goal-directed navigation in simulation. We report the success rate over $25$ navigation trials in four town-weather combinations. The models have been trained in Town 1 and Weather 1. The evaluated models are: img2ctrl -- predicting low-level control from color images; img2wp -- predicting waypoints from color images; seg2ctrl -- predicting low-level control from the segmentation produced by the perception module; ours -- predicting waypoints from the segmentation produced by the perception module. Suffix '+' denotes models trained with data augmentation, and '+dr' denotes the model trained with domain randomization.
• Figure 5: Quantitative evaluation of road following in the real world. We report the average success rate over a total of $11$ navigation trials, with distance to be driven varying from $10$ to $50$ meters. Notation follows Figure \ref{['fig:sim_results']}.