Towards Knowledge-driven Autonomous Driving

TL;DR

The paper argues that data-driven autonomous driving suffers from data bias, long-tail events, and limited interpretability. It advocates a knowledge-driven paradigm that encodes domain knowledge, common sense, and lifelong learning into datasets, environments, and driver agents, leveraging LLMs, world models, and neural rendering. It surveys datasets, benchmarks, simulation tools, and driver-agent architectures, and presents a generalized framework integrating cognition, memory, planning, and reflection to enable robust, cross-domain driving. The work aims to guide research and practice toward safer, more reliable autonomous systems and highlights open-source resources for knowledge-driven development.

Abstract

This paper explores the emerging knowledge-driven autonomous driving technologies. Our investigation highlights the limitations of current autonomous driving systems, in particular their sensitivity to data bias, difficulty in handling long-tail scenarios, and lack of interpretability. Conversely, knowledge-driven methods with the abilities of cognition, generalization and life-long learning emerge as a promising way to overcome these challenges. This paper delves into the essence of knowledge-driven autonomous driving and examines its core components: dataset \& benchmark, environment, and driver agent. By leveraging large language models, world models, neural rendering, and other advanced artificial intelligence techniques, these components collectively contribute to a more holistic, adaptive, and intelligent autonomous driving system. The paper systematically organizes and reviews previous research efforts in this area, and provides insights and guidance for future research and practical applications of autonomous driving. We will continually share the latest updates on cutting-edge developments in knowledge-driven autonomous driving along with the relevant valuable open-source resources at: \url{https://github.com/PJLab-ADG/awesome-knowledge-driven-AD}.

Figures (7)

• Figure 1: Comparison of three technical paradigms to autonomous driving. (1) The rule-based paradigm utilizes the understanding of driving scenarios that are summarized in the scenario semantic space to guide driving. (2) The data-based paradigm tends to model the driving scenarios into the representation space, which is subsequently inferred to the real world to accomplish driving tasks. (3) The knowledge-driven paradigm induces information of driving scenarios into knowledge-augmented representation space, which can be deduced to generalized knowledge in the scenario semantic space, subsequently inferring the scenarios to guide the drive with the knowledge reflection.
• Figure 2: Comparison between the single-domain data-driven paradigm (left), cross-domain data-driven paradigm (center), and the knowledge-driven paradigm (right). The gray $\times$ in the driving scenario represents corner cases, while it transitions to green $\times$, indicating that the method can handle them respectively. Data-driven approaches focus on collecting domain-specific data $d_i$ and obtaining driving capabilities $c_i$ that are limited to handling only similar or corresponding domains $d_i'$. Even if implementing multiple domains data-driven approaches, it only can learn the driving capability $C'$ for processing the union of datasets $D'$. In contrast, knowledge-driven approaches aim to understand coherent features across domains by incorporating human knowledge or common sense and to establish relationships between features, which achieve a broader range of driving capabilities $\hat{C}$ that far exceed the performances of single-domain data-driven and cross-domain data-driven methods, i.e., $\hat{D}\gg D' > \{d_1, d_2, \ldots, d_n\}$.
• Figure 3: Key components in knowledge-driven autonomous driving.
• Figure 4: (a) Traditional and (b) knowledge-augmented autonomous driving datasets. The arrow indicates that the knowledge-augmented datasets are derived from the corresponding source dataset through secondary annotation.
• Figure 5: From the real-world environment to the virtual simulation environment. The utilization of graphics engines enables the perception of real-world environments and the assembly of virtual simulated environments, while this approach incurs high costs. Implicit reconstruction methods, which render simulated environments by collecting data from multiple sources, emerge as a promising and cost-effective solution. Integrating knowledge and data to construct world models facilitates a genuine understanding of the environment, enabling the accomplishment of diverse tasks, particularly in synthesizing data to support closed-loop simulations.