See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers

TL;DR

This work introduces See2Refine, a closed loop framework that uses Vision-Language Model (VLM) perceptual feedback to autonomously design and refine Large Language Model (LLM) based eHMI action designers for external vehicle interfaces. By generating diverse traffic scenarios, rendering action videos, and evaluating them with VLMs, See2Refine creates a kernel score to guide iterative fine tuning via format aware training and Direct Preference Optimization. The approach produces DesignerLLM models that outperform a baseline initial action database across modalities (lightbar, eyes, arm) and align closely with human preferences, while remaining scalable and cost efficient compared to large commercial models. The work also demonstrates cross modality robustness, and shows that VLM guided improvements translate into meaningful enhancements in human perceptual judgments, indicating practical potential for scalable eHMI action design in autonomous systems.

Abstract

Automated vehicles lack natural communication channels with other road users, making external Human-Machine Interfaces (eHMIs) essential for conveying intent and maintaining trust in shared environments. However, most eHMI studies rely on developer-crafted message-action pairs, which are difficult to adapt to diverse and dynamic traffic contexts. A promising alternative is to use Large Language Models (LLMs) as action designers that generate context-conditioned eHMI actions, yet such designers lack perceptual verification and typically depend on fixed prompts or costly human-annotated feedback for improvement. We present See2Refine, a human-free, closed-loop framework that uses vision-language model (VLM) perceptual evaluation as automated visual feedback to improve an LLM-based eHMI action designer. Given a driving context and a candidate eHMI action, the VLM evaluates the perceived appropriateness of the action, and this feedback is used to iteratively revise the designer's outputs, enabling systematic refinement without human supervision. We evaluate our framework across three eHMI modalities (lightbar, eyes, and arm) and multiple LLM model sizes. Across settings, our framework consistently outperforms prompt-only LLM designers and manually specified baselines in both VLM-based metrics and human-subject evaluations. Results further indicate that the improvements generalize across modalities and that VLM evaluations are well aligned with human preferences, supporting the robustness and effectiveness of See2Refine for scalable action design.

Figures (7)

• Figure 1: See2Refine uses VLM-based perceptual evaluation as automated visual feedback to design, evaluate, and iteratively refine LLM-based eHMI action designers without human supervision. In contrast, standard LLM-based designers rely on static prompts and lack perceptual grounding for improvement.
• Figure 2: Our See2Refine framework includes: scenario and action database construction (Section \ref{['method:data']}), format-aware fine-tuning (Section \ref{['method:sft']}), and iterative preference-based learning (Section \ref{['method:dpo']}). A shared action database supports all three components, storing generated actions and expanding to enhance DesignerLLM’s performance.
• Figure 3: Overview of the six eHMI 3D models combining three modalities (lightbar, eyes, arm) and two emitter types (self-driving car, delivery robot). Rendered videos are generated in Blender from the message receivers’ perspective under a defined camera direction and distance.
• Figure 4: Human ratings of five models on six subjective metrics for (a) eyes and (b) lightbar. Bars show mean scores (1--9) with error bars; mental workload is rescaled from 1--20 to 1--9. Ranks are shown above bars (#1 best). DesignerLLM (7B) ranks 4th for both modalities, outperforming the base model and approaching commercial models. $\dagger$ denotes a simple eHMI description; $*$ denotes an enhanced prompt adapted from xia2025automating.
• Figure 5: The layout for each video assessment is in two stages. In the first stage, participants rated their initial impressions of the eHMI using four metrics, without any reference to the intended messages. In the second stage, after the intended messages were revealed, participants re-evaluated their impressions using two different metrics.