Mind the Gap: Evaluating LLM Understanding of Human-Taught Road Safety Principles

TL;DR

The paper addresses the problem of whether multimodal LLMs can understand human-taught road safety concepts. It introduces the Road Safety Understanding Task, implemented with schematic textbook imagery across three categories, evaluated in a zero-shot setting. Findings show partial competence in sign detection and hazard identification but clear gaps in comparative reasoning and schematic abstraction, with performance varying by model and prompting. The work highlights important gaps between human safety education and AI interpretation, underscoring the need for safety-aware ITS and human-centered urban design improvements.

Abstract

Following road safety norms is non-negotiable not only for humans but also for the AI systems that govern autonomous vehicles. In this work, we evaluate how well multi-modal large language models (LLMs) understand road safety concepts, specifically through schematic and illustrative representations. We curate a pilot dataset of images depicting traffic signs and road-safety norms sourced from school text books and use it to evaluate models capabilities in a zero-shot setting. Our preliminary results show that these models struggle with safety reasoning and reveal gaps between human learning and model interpretation. We further provide an analysis of these performance gaps for future research.

Figures (1)

• Figure 1: Road Safety Understanding Task Categories: Sign Detection, Road-Safety Hazard, and Hazard-Pair Comparison, each formatted as multiple-choice questions for zero-shot evaluation.