Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving

Qian Cheng, Weitao Zhou, Cheng Jing, Nanshan Deng, Junze Wen, Zhaoyang Liu, Kun Jiang, Diange Yang

TL;DR

LSRE addresses the gap in enforcing semantic safety for autonomous driving by encoding language-defined rules into a latent, recurrent framework. It uses sparse VLM supervision to train a margin-based latent risk classifier with short-horizon rollouts and hysteresis smoothing, achieving real-time 10 Hz inference without per-frame VLM queries. In CARLA experiments, LSRE matches VLM-level semantic accuracy while delivering earlier hazard anticipation and substantially lower latency, and it generalizes to semantically similar unseen scenes. The approach provides a deployable semantic-safety layer that complements geometric safety and planning in autonomous driving systems.

Abstract

Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-language models (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time deployment.This work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.

LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving

TL;DR

LSRE addresses the gap in enforcing semantic safety for autonomous driving by encoding language-defined rules into a latent, recurrent framework. It uses sparse VLM supervision to train a margin-based latent risk classifier with short-horizon rollouts and hysteresis smoothing, achieving real-time 10 Hz inference without per-frame VLM queries. In CARLA experiments, LSRE matches VLM-level semantic accuracy while delivering earlier hazard anticipation and substantially lower latency, and it generalizes to semantically similar unseen scenes. The approach provides a deployable semantic-safety layer that complements geometric safety and planning in autonomous driving systems.

Abstract

Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-language models (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time deployment.This work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.
Paper Structure (31 sections, 14 equations, 3 figures, 4 tables)

This paper contains 31 sections, 14 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Safety Constraints and Shields in Autonomous Driving
  4. VLM-Based Risk Assessment and Driving Semantics
  5. World Models for Driving and Latent Monitoring
  6. Problem Formulation
  7. Method
  8. System Overview
  9. VLM-Guided Semantic Supervision
  10. Semantic Risk Scoring Module
  11. Latent dynamics
  12. Margin-Based Semantic Risk Classifier
  13. Future Semantic Value Estimation
  14. Hysteresis-Based Temporal Smoothing
  15. Experiment
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Overall pipeline of LSRE. A pretrained vision--language model (VLM) provides sparse semantic-risk supervision for key frames. A recurrent state-space world model encodes multi-view observations into latent states with temporal dynamics, and generates short-horizon rollouts. A lightweight latent classifier, trained under VLM supervision, evaluates both instantaneous and predicted future latent states to produce a real-time semantic risk signal for the driving stack.
  • Figure 2: Three semantic–failure categories used in evaluation, each instantiated with two scenario variants (in–distribution and few–shot). These scenarios capture human-understandable but rule-hard safety semantics that are essential for evaluating real-time semantic risk detection.
  • Figure 3: The semantic risk value output by the proposed LSRE model across three example cases. Each subplot highlights the LSRE detection time and the ground-truth (GT) risk happened time.