Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Driving

TL;DR

FM4SU tackles the challenge of understanding spatio-temporal driving scenes by building a symbolic BEV representation from the nuScenesKG knowledge graph and training a transformer-based foundation model on serialized token sequences. The approach leverages a two-part learning objective—Scene Object Prediction and Next Scene Prediction—grounded in an ontology-driven representation $S_T=(B,C,T,P)$ and a KG structure $G=(E,R,E|V)$, achieving high accuracy on both tasks ($88.7\%$ and $86.7\%$, respectively). Results show strong gains over zero-shot baselines and demonstrate the value of explicit knowledge and scene priors for robust scene understanding, with metadata and model size affecting performance. The work provides a BEV symbolic scene dataset and open-source tooling, and points to further integration with additional LLMs and downstream tasks such as 3D object detection and trajectory forecasting.

Abstract

The autonomous driving field has seen remarkable advancements in various topics, such as object recognition, trajectory prediction, and motion planning. However, current approaches face limitations in effectively comprehending the complex evolutions of driving scenes over time. This paper proposes FM4SU, a novel methodology for training a symbolic foundation model (FM) for scene understanding in autonomous driving. It leverages knowledge graphs (KGs) to capture sensory observation along with domain knowledge such as road topology, traffic rules, or complex interactions between traffic participants. A bird's eye view (BEV) symbolic representation is extracted from the KG for each driving scene, including the spatio-temporal information among the objects across the scenes. The BEV representation is serialized into a sequence of tokens and given to pre-trained language models (PLMs) for learning an inherent understanding of the co-occurrence among driving scene elements and generating predictions on the next scenes. We conducted a number of experiments using the nuScenes dataset and KG in various scenarios. The results demonstrate that fine-tuned models achieve significantly higher accuracy in all tasks. The fine-tuned T5 model achieved a next scene prediction accuracy of 86.7%. This paper concludes that FM4SU offers a promising foundation for developing more comprehensive models for scene understanding in autonomous driving.

Figures (9)

• Figure 1: Motivation -- An exemplary driving scene illustrating how FM4SU is able to enrich scene understanding. By learning spatio-temporal patterns within and across diverse driving scenarios, FM4SU can infer missing information (e.g. pedestrian crossing). This enables more informed and consequently safer decision-making for the vehicle.
• Figure 2: The Learning Pipeline - comprises four main phases: 1) Perception - captures the information from the environment using various sensors; 2) Knowledge Graph Representation - structures information in the form of entities and relations including rich semantics and domain knowledge; 3) Bird-Eye-View Representation - extracts and transforms information in a matrix representation; and 4) Scene Learning - learns scene evolution based on the co-occurring elements.
• Figure 3: nuScenes Knowledge Graph - a comprehensive representation of sensory data using ontological concepts with a strong focus on scene understanding. Scene objects $O$ are categorized based on different levels of abstraction and characteristics, such as static and dynamic for fixed and moving objects, respectively.
• Figure 4: (a) A matrix of cells with scene objects is extracted from the driving scenes KG. (b) The matrices extracted from the scenes at time steps $T$ and $T+1$ are converted into serialized sequences of tokens. The language model is trained using either (c) scene object prediction or (d) next scene prediction.
• Figure 5: (a) A nuScenes visualization of a traffic scene using top-down LiDAR data, representing different entities with distinct colors. (b) A BEV representation of map concepts, including lanes, sidewalks, etc., is depicted as polygons. A $20 \times 11$ area matrix $A$ to illustrate the scene range in our setup. The pinpoint indicates the EV’s location.