RAG-based Explainable Prediction of Road Users Behaviors for Automated Driving using Knowledge Graphs and Large Language Models

TL;DR

The paper tackles the challenge of predicting road users' behaviors in autonomous driving with contextual explainability. It presents a neuro-symbolic framework that combines Knowledge Graphs and Bayesian inference, enhanced by Retrieval Augmented Generation to provide human-readable explanations. Two use cases—pedestrian crossing actions and vehicle lane-change maneuvers—are developed using PedFeatKG and PedFeatRulesKG ontologies, with evaluations showing improvements in anticipation and F1 over state-of-the-art baselines. The approach demonstrates strong explainability through fuzzy-rule integration and RAG, signaling a promising path toward human-understandable and safe autonomous driving systems that can generalize across diverse contexts and knowledge sources.

Abstract

Prediction of road users' behaviors in the context of autonomous driving has gained considerable attention by the scientific community in the last years. Most works focus on predicting behaviors based on kinematic information alone, a simplification of the reality since road users are humans, and as such they are highly influenced by their surrounding context. In addition, a large plethora of research works rely on powerful Deep Learning techniques, which exhibit high performance metrics in prediction tasks but may lack the ability to fully understand and exploit the contextual semantic information contained in the road scene, not to mention their inability to provide explainable predictions that can be understood by humans. In this work, we propose an explainable road users' behavior prediction system that integrates the reasoning abilities of Knowledge Graphs (KG) and the expressiveness capabilities of Large Language Models (LLM) by using Retrieval Augmented Generation (RAG) techniques. For that purpose, Knowledge Graph Embeddings (KGE) and Bayesian inference are combined to allow the deployment of a fully inductive reasoning system that enables the issuing of predictions that rely on legacy information contained in the graph as well as on current evidence gathered in real time by onboard sensors. Two use cases have been implemented following the proposed approach: 1) Prediction of pedestrians' crossing actions; 2) Prediction of lane change maneuvers. In both cases, the performance attained surpasses the current state of the art in terms of anticipation and F1-score, showing a promising avenue for future research in this field.

Figures (11)

• Figure 1: The target vehicle (green) will most likely make a left lane change maneuver based on the risk assessment of the surrounding (blue) vehicles.
• Figure 2: \ref{['fig:Generated Lane Change KG Instance']} One KG instance where the vehicle has zero lateral acceleration and has medium TTC risk with the preceding vehicle and high TTC with the left following vehicle. (b) PedFeatKG from explainable features with 1 instance.
• Figure 3: Pipeline architecture for modelling road user's behaviors.
• Figure 4: Fuzzy rule conversion definition.
• Figure 5: PedFeatRulesKG from explainable features with 1 instance.