Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

Ji Zhou, Yilin Ding, Yongqi Zhao, Jiachen Xu, Arno Eichberger

TL;DR

The paper investigates whether Large Vision-Language Models (LVLMs) can support 2D object detection under Safety of the Intended Functionality (SOTIF) conditions by benchmarking ten LVLMs on the PeSOTIF dataset against a YOLO baseline. It introduces a unified visual-grounding pipeline with structured prompts and a JSON-based output parser to obtain bounding-box predictions without task-specific fine-tuning. The results show that top LVLMs achieve higher recall and robustness to visual degradation than the YOLO baseline in natural and adverse conditions, while conventional detectors retain higher geometric precision on crafted perturbations; latency remains a major constraint. The study underscores the potential of LVLMs as high-level safety validators in hybrid perception systems and provides a reproducible benchmark to guide future research toward lightweight distillation and improved spatial grounding for real-time deployment.

Abstract

Reliable environmental perception remains one of the main obstacles for safe operation of automated vehicles. Safety of the Intended Functionality (SOTIF) concerns safety risks from perception insufficiencies, particularly under adverse conditions where conventional detectors often falter. While Large Vision-Language Models (LVLMs) demonstrate promising semantic reasoning, their quantitative effectiveness for safety-critical 2D object detection is underexplored. This paper presents a systematic evaluation of ten representative LVLMs using the PeSOTIF dataset, a benchmark specifically curated for long-tail traffic scenarios and environmental degradations. Performance is quantitatively compared against the classical perception approach, a YOLO-based detector. Experimental results reveal a critical trade-off: top-performing LVLMs (e.g., Gemini 3, Doubao) surpass the YOLO baseline in recall by over 25% in complex natural scenarios, exhibiting superior robustness to visual degradation. Conversely, the baseline retains an advantage in geometric precision for synthetic perturbations. These findings highlight the complementary strengths of semantic reasoning versus geometric regression, supporting the use of LVLMs as high-level safety validators in SOTIF-oriented automated driving systems.

A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

TL;DR

The paper investigates whether Large Vision-Language Models (LVLMs) can support 2D object detection under Safety of the Intended Functionality (SOTIF) conditions by benchmarking ten LVLMs on the PeSOTIF dataset against a YOLO baseline. It introduces a unified visual-grounding pipeline with structured prompts and a JSON-based output parser to obtain bounding-box predictions without task-specific fine-tuning. The results show that top LVLMs achieve higher recall and robustness to visual degradation than the YOLO baseline in natural and adverse conditions, while conventional detectors retain higher geometric precision on crafted perturbations; latency remains a major constraint. The study underscores the potential of LVLMs as high-level safety validators in hybrid perception systems and provides a reproducible benchmark to guide future research toward lightweight distillation and improved spatial grounding for real-time deployment.

Abstract

Reliable environmental perception remains one of the main obstacles for safe operation of automated vehicles. Safety of the Intended Functionality (SOTIF) concerns safety risks from perception insufficiencies, particularly under adverse conditions where conventional detectors often falter. While Large Vision-Language Models (LVLMs) demonstrate promising semantic reasoning, their quantitative effectiveness for safety-critical 2D object detection is underexplored. This paper presents a systematic evaluation of ten representative LVLMs using the PeSOTIF dataset, a benchmark specifically curated for long-tail traffic scenarios and environmental degradations. Performance is quantitatively compared against the classical perception approach, a YOLO-based detector. Experimental results reveal a critical trade-off: top-performing LVLMs (e.g., Gemini 3, Doubao) surpass the YOLO baseline in recall by over 25% in complex natural scenarios, exhibiting superior robustness to visual degradation. Conversely, the baseline retains an advantage in geometric precision for synthetic perturbations. These findings highlight the complementary strengths of semantic reasoning versus geometric regression, supporting the use of LVLMs as high-level safety validators in SOTIF-oriented automated driving systems.
Paper Structure (20 sections, 4 equations, 11 figures, 1 table)

This paper contains 20 sections, 4 equations, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. PeSOTIF Dataset
  4. Image Preprocessing
  5. Image Resize
  6. Draw Border
  7. Draw Scale
  8. LVLMs Under Test
  9. LVLMs Specification
  10. Prompt Engineering
  11. LVLM Output Parsing
  12. Performance Evaluation
  13. Result
  14. Overall Performance
  15. Performance in Environment Subset
  16. ...and 5 more sections

Figures (11)

  • Figure 1: An example of potential perception functional insufficiency under severe glare from oncoming traffic, as provided in the PeSOTIF dataset peng2023pesotif.
  • Figure 2: Overall workflow of the LVLM-based 2D object recognition in PeSOTIF dataset.
  • Figure 3: Overview of PeSOTIF dataset peng2023pesotif.
  • Figure 4: Structure of the LLM prompt for 2D object detection, consisting of (a) role definition, (b) task specification with class taxonomy and coordinate reference, (c) output format specification in JSON, and (d) constraints to enforce format consistency.
  • Figure 5: Visualization of ten LVLMs' performance in PeSOTIF dataset.
  • ...and 6 more figures