KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System

TL;DR

KnowVal addresses the need for safe, interpretable, and value-aligned autonomous driving in open-world contexts by integrating perception-guided knowledge retrieval with a driving knowledge graph and a dedicated Value Model. It constructs a comprehensive knowledge graph capturing traffic laws, defensive driving principles, and ethical norms, and uses an LLM-based retrieval mechanism grounded in perceptual observations. A Value Model trained on a human-preference dataset evaluates trajectory compliance with retrieved rules, enabling value-guided planning that combines world-state prediction with knowledge-based scoring. Experimental results on nuScenes and Bench2Drive show improved safety (lowest collision rate on nuScenes) and state-of-the-art performance on Bench2Drive, while remaining compatible with existing architectures and planning paradigms.

Abstract

Visual-language reasoning, driving knowledge, and value alignment are essential for advanced autonomous driving systems. However, existing approaches largely rely on data-driven learning, making it difficult to capture the complex logic underlying decision-making through imitation or limited reinforcement rewards. To address this, we propose KnowVal, a new autonomous driving system that enables visual-language reasoning through the synergistic integration of open-world perception and knowledge retrieval. Specifically, we construct a comprehensive driving knowledge graph that encodes traffic laws, defensive driving principles, and ethical norms, complemented by an efficient LLM-based retrieval mechanism tailored for driving scenarios. Furthermore, we develop a human-preference dataset and train a Value Model to guide interpretable, value-aligned trajectory assessment. Experimental results show that our method substantially improves planning performance while remaining compatible with existing architectures. Notably, KnowVal achieves the lowest collision rate on nuScenes and state-of-the-art results on Bench2Drive.

Figures (8)

• Figure 1: Comparison of End-to-End, VLA, and our KnowVal. Our 3D vision system enables visual–language reasoning through the mutual guidance of perception and retrieval, understands law and morality via knowledge retrieval, and integrates values through a dedicated value model.
• Figure 2: Overall architecture of KnowVal.KnowVal first performs retrieval-guided open-world perception to extract instance-, panoramic-, and concept-level information. It then conducts perception-guided retrieval from the knowledge graph to obtain relevant knowledge. Finally, decision-making is achieved through integrated planning, world prediction, and value assessment. KnowVal is compatible with existing methods. The Specialized Perception, Planning Model, and World Model are obtained by modifying and fine-tuning existing methods. Feature propagation across modules ensures comprehensive information flow throughout the system.
• Figure 3: Pre-construction of the knowledge graph and retrieval process in KnowVal. We collect diverse driving-related resources—including laws, regulations, defensive driving principles, moral guidelines, and experiential knowledge—to construct an initial knowledge forest based on textual structures. Large language models (LLMs) are then used to extract entities and define vertices and edges, forming a structured knowledge graph. During inference, KnowVal generates queries enriched with 3D perception information to retrieve and rank relevant entries from the knowledge graph by descending relevance.
• Figure 4: Planning process of KnowVal. We design a trajectory generation method compatible with existing Transformer- and RNN-based planners. By combining future world state prediction with value model assessment, KnowVal produces diverse candidate trajectories and makes interpretable final decisions.
• Figure 5: Illustration of the inference process of KnowVal.KnowVal perceives long-tail scenarios, retrieves ethical knowledge related to pedestrians and puddles, and employs the Value Model to select appropriate actions.