nuScenes Knowledge Graph -- A comprehensive semantic representation of traffic scenes for trajectory prediction

TL;DR

This work addresses the need for rich, semantically grounded scene representations to improve trajectory prediction in autonomous driving. It introduces a rigorous traffic-scene ontology and constructs the nuScenes Knowledge Graph ($nSKG$) to model agents, map elements, and spatio-temporal relations, plus a PyG-ready heterogeneous scene-graph dataset ($nSTP$) for training graph neural networks. The approach enables a neuro-symbolic AI workflow by providing both symbolic knowledge (KG) and sub-symbolic training data, with local-coordinate invariance mechanisms to enhance generalization. The resulting resources offer a comprehensive, scalable representation that can improve robustness and safety in trajectory prediction and enable research into richer map and interaction information beyond prior datasets.

Abstract

Trajectory prediction in traffic scenes involves accurately forecasting the behaviour of surrounding vehicles. To achieve this objective it is crucial to consider contextual information, including the driving path of vehicles, road topology, lane dividers, and traffic rules. Although studies demonstrated the potential of leveraging heterogeneous context for improving trajectory prediction, state-of-the-art deep learning approaches still rely on a limited subset of this information. This is mainly due to the limited availability of comprehensive representations. This paper presents an approach that utilizes knowledge graphs to model the diverse entities and their semantic connections within traffic scenes. Further, we present nuScenes Knowledge Graph (nSKG), a knowledge graph for the nuScenes dataset, that models explicitly all scene participants and road elements, as well as their semantic and spatial relationships. To facilitate the usage of the nSKG via graph neural networks for trajectory prediction, we provide the data in a format, ready-to-use by the PyG library. All artefacts can be found here: https://github.com/boschresearch/nuScenes_Knowledge_Graph

Figures (5)

• Figure 1: We model traffic scenes (top left) by applying a rigorous ontology (bottom left) to them, producing rich, temporal, heterogeneous graphs. We provide a large graph regression dataset of $(x_i, y_i)$ pairs for training GNNs on the designed representation. Partial image credits: https://www.freepik.com/free-vector/global-communication-background-business-network-vector-design_19585255.htm.
• Figure 2: Model of the temporal nature of traffic scenarios applied to a single car travelling along a lane.
• Figure 3: Semantic relationship model between agents.
• Figure 4: Example of how isOn models the spatial relation between agents and map elements. In addition, the given scenario illustrates stop areas and lane snippets.
• Figure 5: An excerpt of our ontology.