HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving

TL;DR

HERMES addresses safety-critical long-tail scenarios in end-to-end autonomous driving by coupling foundation-model–driven long-tail semantics with a risk-aware trajectory planner. A teacher module produces Long-Tail Scene Context and Long-Tail Planning Context via prompt-engineered VLM annotations, which are encoded into embeddings and fed to a Tri-Modal Driving Module that fuses multi-view vision, history, and semantic guidance through cross-attention and intent conditioning. The approach demonstrates superior safety-aligned planning on the WOD-E2E-based long-tail benchmark, with notable gains in the RFS metric and robust performance across diverse scenario categories, validated through ablations and qualitative analyses. This work advances practical deployment by bridging semantic reasoning with real-time control, enabling safer navigation in heterogeneous and rare driving events, and suggests avenues for richer long-tail annotation pipelines and safer closed-loop evaluation.

Abstract

End-to-end autonomous driving models increasingly benefit from large vision--language models for semantic understanding, yet ensuring safe and accurate operation under long-tail conditions remains challenging. These challenges are particularly prominent in long-tail mixed-traffic scenarios, where autonomous vehicles must interact with heterogeneous road users, including human-driven vehicles and vulnerable road users, under complex and uncertain conditions. This paper proposes HERMES, a holistic risk-aware end-to-end multimodal driving framework designed to inject explicit long-tail risk cues into trajectory planning. HERMES employs a foundation-model-assisted annotation pipeline to produce structured Long-Tail Scene Context and Long-Tail Planning Context, capturing hazard-centric cues together with maneuver intent and safety preference, and uses these signals to guide end-to-end planning. HERMES further introduces a Tri-Modal Driving Module that fuses multi-view perception, historical motion cues, and semantic guidance, ensuring risk-aware accurate trajectory planning under long-tail scenarios. Experiments on the real-world long-tail dataset demonstrate that HERMES consistently outperforms representative end-to-end and VLM-driven baselines under long-tail mixed-traffic scenarios. Ablation studies verify the complementary contributions of key components.

Figures (8)

• Figure 1: Comparison of autonomous driving paradigms. (a) Traditional end-to-end models integrate perception, prediction, and planning in a unified pipeline. (b) VLM/VLA-based approaches leverage foundation models for reasoning and planning. (c) HERMES (ours) combines VLM-generated long-tail context embeddings with a risk-aware end-to-end driving model for safe trajectory planning.
• Figure 2: Overview of the HERMES framework. The Long-Tail Instruction Embedding module employs a cloud-based VLM (e.g., Qwen-VL-Flash) to analyze 8-camera images and generate structured annotations comprising Long-Tail Scene Context and Long-Tail Planning Context, which are encoded by BGE-M3 into text embeddings. The Tri-Modal Driving Module sequentially processes multimodal inputs: the Vision Encoder first extracts spatial features, which are fused with scene embeddings via Scene Fusion to produce scene context; subsequently, the State Encoder processes historical trajectories, which are integrated with scene context through Planning Context Fusion; finally, Intent Modulator, Risk Planning Cross-Attention, and Temporal Decoder generate risk-aware trajectories by balancing VLM guidance with learned trajectory patterns.
• Figure 3: Architecture of the Intent Modulator. High-level driving intent is embedded and processed through a two-layer MLP to generate scale and shift parameters, which adaptively modulate the planning context.
• Figure 4: Architecture of the Risk Planning Cross-Attention module. The intent-aware planning context attends to the planning instruction embedding through multi-head cross-attention, and the resulting semantic adjustment is integrated via a scaled residual connection followed by layer normalization.
• Figure 5: Nighttime driving under heavy rain and poor visibility. Planned trajectory is shown in red.