Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning

TL;DR

This paper proposes an automatic curriculum learning framework that dynamically generates driving scenarios with adaptive complexity based on the agent's evolving capabilities and incorporates a “teacher” that automatically generates and mutates driving scenarios based on their learning potential, eliminating the need for expert design.

Abstract

This paper addresses the challenges of training end-to-end autonomous driving agents using Reinforcement Learning (RL). RL agents are typically trained in a fixed set of scenarios and nominal behavior of surrounding road users in simulations, limiting their generalization and real-life deployment. While domain randomization offers a potential solution by randomly sampling driving scenarios, it frequently results in inefficient training and sub-optimal policies due to the high variance among training scenarios. To address these limitations, we propose an automatic curriculum learning framework that dynamically generates driving scenarios with adaptive complexity based on the agent's evolving capabilities. Unlike manually designed curricula that introduce expert bias and lack scalability, our framework incorporates a ``teacher'' that automatically generates and mutates driving scenarios based on their learning potential -- an agent-centric metric derived from the agent's current policy -- eliminating the need for expert design. The framework enhances training efficiency by excluding scenarios the agent has mastered or finds too challenging. We evaluate our framework in a reinforcement learning setting where the agent learns a driving policy from camera images. Comparative results against baseline methods, including fixed scenario training and domain randomization, demonstrate that our approach leads to enhanced generalization, achieving higher success rates: +9% in low traffic density, +21% in high traffic density, and faster convergence with fewer training steps. Our findings highlight the potential of ACL in improving the robustness and efficiency of RL-based autonomous driving agents.

Figures (4)

• Figure 1: The proposed framework alternates between two modes based on the replay decision $d$. When $d = 0$, a random generator creates diverse scenarios by varying environment parameters. Scenarios with high learning potential are conditionally added to the scenario buffer $\Lambda$, ensuring training efficiency. When $d = 1$, the student trains exclusively on scenarios sampled from $\Lambda$, while an editor mutates them to further refine the most effective scenarios for enhancing the student's learning progress.
• Figure 2: A driving scenario represented as a directed graph, visualized from a Bird’s Eye View perspective. Nodes are sampled at equidistant intervals along the road and can be occupied by the student, NPCs, obstacles, or remain empty. Edges define the road topology and the goal destinations for the student and NPCs.
• Figure 3: A comparison between the average number of actors in the training scenarios generated by DR (red) and the proposed ACL framework (blue). The figure illustrates that our framework can generate a curriculum with an evolving complexity compared to random scenarios from DR.
• Figure 4: Exemplary scenarios generated by the proposed ACL framework at different phases of the student training. It can be observed that the scenarios have emerging complexity as the training progresses.