A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving

TL;DR

A robust Deep Reinforcement Learning (DRL) framework that incorporates platform-dependent perception modules to extract task-relevant information, enabling the training of a lane-following and overtaking agent in simulation to facilitate the efficient transfer of the DRL agent to new simulated environments and the real world with minimal adjustments.

Abstract

Deep Reinforcement Learning (DRL) has shown remarkable success in solving complex tasks across various research fields. However, transferring DRL agents to the real world is still challenging due to the significant discrepancies between simulation and reality. To address this issue, we propose a robust DRL framework that leverages platform-dependent perception modules to extract task-relevant information and train a lane-following and overtaking agent in simulation. This framework facilitates the seamless transfer of the DRL agent to new simulated environments and the real world with minimal effort. We evaluate the performance of the agent in various driving scenarios in both simulation and the real world, and compare it to human players and the PID baseline in simulation. Our proposed framework significantly reduces the gaps between different platforms and the Sim2Real gap, enabling the trained agent to achieve similar performance in both simulation and the real world, driving the vehicle effectively.

Figures (11)

• Figure 1: Robust Sim2Real transfer with DRL agent and platform-dependent perception module that separates the agent from the environment. The DRL agent is first trained in the Gazebo simulator with the information provided by the perception module in Gazebo, afterwards, the trained agent is evaluated in Gym-Duckietown environment, and the real-world scenario with real Duckiebots without additional effort but only with platform specified parameterized perception module. During the training process, two different driving scenarios for the agent to tackle, are lane following (Scenario 1) and overtaking (Scenario 2) tasks.
• Figure 2: Evaluation results in Gym-Duckietown. a, Illustration of sampled vehicle trajectories for different approaches within five different maps. b, Boxplots of different performance metrics for different approaches during the evaluation process. c, Final scores comparison between the DRL agent and other baselines, where the fast-mode DRL agent achieves the highest score in every evaluation map and is followed by the best human player, slow-mode DRL agent, and PID baseline.
• Figure 3: Evaluation results of different approaches for lane following in real-world scenarios. a, Illustration of the vehicle trajectories for the outer ring of five different maps. b, Trajectories of different approaches running on the inner ring. ${}^\star$ indicates the proposed DRL agent runs on DB19.
• Figure 4: Evaluation results for lane following in real-world scenario for one vehicle, the other two are shown in Fig. \ref{['fig:real-world lane following vehicle 23']}. a, Illustration of the vehicle trajectories for different approaches i.e. DRL agent and PID baseline on the inner and outer ring of the real-world track. b, Distribution histogram and kernel density estimate (KDE) plots of the lateral and orientation deviation for PID baseline and DRL agent during the real-world lane following evaluation.
• Figure 5: Evaluation results for real-world overtaking scenarios. The first row depicts the trajectories of the proposed agent as it successfully navigates around three static vehicles (with red rectangles). The second row illustrates the agent overtaking a slower vehicle, shown in light red, with red circles marking the exact positions where the overtaking maneuvers occurred.