Exploring the Limitations of Behavior Cloning for Autonomous Driving

TL;DR

This paper investigates end-to-end behavior cloning for autonomous driving using a new NoCrash benchmark in the CARLA simulator. It introduces a strong off-policy Conditional Imitation Learning baseline (CILRS) with a deeper ResNet backbone and a speed-prediction branch, achieving state-of-the-art results in complex urban scenarios and unseen environments. The study systematically reveals limitations of behavior cloning, including dataset bias, causal confusion, inertia, and high training variance, especially in dynamic traffic, and proposes NoCrash to better assess reactions to dynamic agents. It demonstrates the need for causal models, data diversity, and improved training stability before end-to-end BC can be considered for real-world deployment.

Abstract

Driving requires reacting to a wide variety of complex environment conditions and agent behaviors. Explicitly modeling each possible scenario is unrealistic. In contrast, imitation learning can, in theory, leverage data from large fleets of human-driven cars. Behavior cloning in particular has been successfully used to learn simple visuomotor policies end-to-end, but scaling to the full spectrum of driving behaviors remains an unsolved problem. In this paper, we propose a new benchmark to experimentally investigate the scalability and limitations of behavior cloning. We show that behavior cloning leads to state-of-the-art results, including in unseen environments, executing complex lateral and longitudinal maneuvers without these reactions being explicitly programmed. However, we confirm well-known limitations (due to dataset bias and overfitting), new generalization issues (due to dynamic objects and the lack of a causal model), and training instability requiring further research before behavior cloning can graduate to real-world driving. The code of the studied behavior cloning approaches can be found at https://github.com/felipecode/coiltraine .

Figures (12)

• Figure 1: Driving scenarios from our new benchmark where the agent needs to react to dynamic changes in the environment, handle clutter (only part of the environment is causally relevant), and predict complex sensorimotor controls (lateral and longitudinal). We show that Behavior Cloning yields state-of-the-art policies in these complex scenarios and investigate its limitations.
• Figure 2: Our proposed network architecture, called CILRS, for end-to-end urban driving based on CIL Codevilla2018. A ResNet perception module processes an input image to a latent space followed by two prediction heads: one for controls and one for speed.
• Figure 3: Due to biases in the data, the results may get either saturated or worse with increasing amounts of training data.
• Figure 4: The percentage of episodes that failed due to the inertia problem. We can see that by increasing the amount of data, this bias may further degrade the generalization capabilities of the models.
• Figure 5: Comparison between the results with and without the speed prediction and different amounts of training demonstrations. We report the results only for the case were highest generalization is needed (New Weather and Town).