Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection

Roberto Brusnicki, David Pop, Yuan Gao, Mattia Piccinini, Johannes Betz

TL;DR

SAVANT tackles the challenge of detecting semantic anomalies in autonomous driving by moving from unstructured vision-language prompting to a structured, four-layer scene analysis and a two-phase pipeline. By combining sequenced textual scene descriptions with multimodal evaluation, SAVANT achieves high recall and accuracy and demonstrates that fine-tuned open-source models can surpass proprietary systems for practical, local deployment. The work provides extensive evaluation across 33 VLMs, introduces DSPy-driven prompt optimization, and delivers a large, richly annotated CODALM dataset to address data scarcity in anomaly detection. These contributions enable safer, more accessible semantic monitoring for real-world autonomous systems and offer a practical path toward low-cost, high-performance anomaly detection at deployment scale.

Abstract

Autonomous driving systems remain critically vulnerable to the long-tail of rare, out-of-distribution scenarios with semantic anomalies. While Vision Language Models (VLMs) offer promising reasoning capabilities, naive prompting approaches yield unreliable performance and depend on expensive proprietary models, limiting practical deployment. We introduce SAVANT (Semantic Analysis with Vision-Augmented Anomaly deTection), a structured reasoning framework that achieves high accuracy and recall in detecting anomalous driving scenarios from input images through layered scene analysis and a two-phase pipeline: structured scene description extraction followed by multi-modal evaluation. Our approach transforms VLM reasoning from ad-hoc prompting to systematic analysis across four semantic layers: Street, Infrastructure, Movable Objects, and Environment. SAVANT achieves 89.6% recall and 88.0% accuracy on real-world driving scenarios, significantly outperforming unstructured baselines. More importantly, we demonstrate that our structured framework enables a fine-tuned 7B parameter open-source model (Qwen2.5VL) to achieve 90.8% recall and 93.8% accuracy - surpassing all models evaluated while enabling local deployment at near-zero cost. By automatically labeling over 9,640 real-world images with high accuracy, SAVANT addresses the critical data scarcity problem in anomaly detection and provides a practical path toward reliable, accessible semantic monitoring for autonomous systems.

SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection

TL;DR

SAVANT tackles the challenge of detecting semantic anomalies in autonomous driving by moving from unstructured vision-language prompting to a structured, four-layer scene analysis and a two-phase pipeline. By combining sequenced textual scene descriptions with multimodal evaluation, SAVANT achieves high recall and accuracy and demonstrates that fine-tuned open-source models can surpass proprietary systems for practical, local deployment. The work provides extensive evaluation across 33 VLMs, introduces DSPy-driven prompt optimization, and delivers a large, richly annotated CODALM dataset to address data scarcity in anomaly detection. These contributions enable safer, more accessible semantic monitoring for real-world autonomous systems and offer a practical path toward low-cost, high-performance anomaly detection at deployment scale.

Abstract

Autonomous driving systems remain critically vulnerable to the long-tail of rare, out-of-distribution scenarios with semantic anomalies. While Vision Language Models (VLMs) offer promising reasoning capabilities, naive prompting approaches yield unreliable performance and depend on expensive proprietary models, limiting practical deployment. We introduce SAVANT (Semantic Analysis with Vision-Augmented Anomaly deTection), a structured reasoning framework that achieves high accuracy and recall in detecting anomalous driving scenarios from input images through layered scene analysis and a two-phase pipeline: structured scene description extraction followed by multi-modal evaluation. Our approach transforms VLM reasoning from ad-hoc prompting to systematic analysis across four semantic layers: Street, Infrastructure, Movable Objects, and Environment. SAVANT achieves 89.6% recall and 88.0% accuracy on real-world driving scenarios, significantly outperforming unstructured baselines. More importantly, we demonstrate that our structured framework enables a fine-tuned 7B parameter open-source model (Qwen2.5VL) to achieve 90.8% recall and 93.8% accuracy - surpassing all models evaluated while enabling local deployment at near-zero cost. By automatically labeling over 9,640 real-world images with high accuracy, SAVANT addresses the critical data scarcity problem in anomaly detection and provides a practical path toward reliable, accessible semantic monitoring for autonomous systems.
Paper Structure (23 sections, 4 equations, 5 figures, 4 tables)

This paper contains 23 sections, 4 equations, 5 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Foundation Models for Driving Intelligence
  4. Detection of Semantic Anomaly and Out-of-Distribution Inputs
  5. Critical Summary
  6. Contributions
  7. THE SAVANT FRAMEWORK
  8. Layered Anomaly Formulation
  9. Two-Phase Anomaly Detection Pipeline
  10. Fine-tuning Integration Strategy
  11. EXPERIMENTAL SETUP
  12. Datasets
  13. Models Evaluated
  14. Evaluation Methods
  15. Evaluation Metrics and Implementation
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Overview of the SAVANT framework. Driving images are processed in two phases: (1) Scene Description Extraction, where a VLM generates textual descriptions of scene layers (street, infrastructure, movable objects, environmental); (2) Scene Evaluation, where the original image and aggregated descriptions are jointly analyzed for anomaly classification. The resulting classifications undergo human verification and correction through Human-in-the-Loop (HITL) curation to create a high-quality training dataset, which is subsequently used to fine-tune an enhanced VLM capable of single-shot anomaly detection while maintaining compatibility with the original multi-phase framework.
  • Figure 2: Examples of semantic anomalies in autonomous driving scenarios. These real-world cases demonstrate contextually unusual situations where individually recognizable objects combine to create potentially unsafe or confusing scenarios for autonomous systems.
  • Figure 3: Layer-wise anomaly distribution comparing CODALM_medium dataset (5,078 samples) and its test split (1020 samples):(a) Individual layer frequency across the four semantic layers, (b) Anomaly layer combination frequency, (c) Multi-layer anomaly frequency distribution.
  • Figure 4: Resolution comparison analysis across different image resolutions for the best-performing models of each family. Performance shows significant improvements up to 360p with diminishing returns (or worse performance) at higher resolutions.
  • Figure 5: Failure rates (%) across semantic layer combinations for three Qwen2.5-VL-7B variants: FT (Fine-Tuned), PFT (Pipeline Fine-Tuned), and NFT (Non-Fine-Tuned). Layer abbreviations: S (Street), I (Infrastructure), M (Movable Objects), E (Environmental). Multi-layer combinations (e.g., E.S.I.M) represent anomalies spanning multiple semantic contexts.