Vision Foundation Model Embedding-Based Semantic Anomaly Detection

TL;DR

The paper addresses semantic anomalies—contextual misconfigurations that challenge high-level reasoning in autonomous systems—by leveraging vision foundation model embeddings directly from images. It proposes a lightweight, embedding-based framework with two variants (grid-based patch embeddings and instance-centric embeddings from OWLv2+SAM2) and a simple filtering step, evaluated on CARLA simulations. Results show that the instance-based approach with filtering can match GPT-4o performance while providing precise anomaly localization, demonstrating the practical potential of vision embeddings for real-time semantic anomaly detection. The work highlights a path toward faster, localization-capable anomaly detection and suggests future directions, including adaptive scoring and graph-based representations to preserve scene context. Such methods promise improved safety in real-world autonomous systems by enabling timely detection and localization of semantically anomalous scenes.

Abstract

Semantic anomalies are contextually invalid or unusual combinations of familiar visual elements that can cause undefined behavior and failures in system-level reasoning for autonomous systems. This work explores semantic anomaly detection by leveraging the semantic priors of state-of-the-art vision foundation models, operating directly on the image. We propose a framework that compares local vision embeddings from runtime images to a database of nominal scenarios in which the autonomous system is deemed safe and performant. In this work, we consider two variants of the proposed framework: one using raw grid-based embeddings, and another leveraging instance segmentation for object-centric representations. To further improve robustness, we introduce a simple filtering mechanism to suppress false positives. Our evaluations on CARLA-simulated anomalies show that the instance-based method with filtering achieves performance comparable to GPT-4o, while providing precise anomaly localization. These results highlight the potential utility of vision embeddings from foundation models for real-time anomaly detection in autonomous systems.

Figures (10)

• Figure 1: Examples of semantic anomalies (a,c) and their CARLA-simulated equivalents (b,d). A truck with traffic lights (a) confused a Tesla into detecting active signals robitzski2021trafficlight. A stop sign on a billboard (c) caused unintended braking kelly2017stopSignrobitzski2021stopsign.
• Figure 2: Overview of the proposed vision-based semantic anomaly detection framework. Structuring semantic anomaly detection in this way enables detecting anomalies without requiring access to out-of-distribution data.
• Figure 3: Qualitative comparison of anomaly detections. Each method (embedding-based left, instance-based right) shows two true positives (top row) and two false positives (bottom row).
• Figure 4: Distribution of anomaly scores for anomalies (T) and nominal objects (N) across scenarios.
• Figure 5: Threshold sweep showing metric trends (IoU, F1, TPR, FPR) for both methods across all scenarios.