On the Potential of Open-Vocabulary Models for Object Detection in Unusual Street Scenes

TL;DR

This work evaluates open-vocabulary object detectors for out-of-distribution (OOD) object detection in unusual street scenes using the OoDIS benchmark and LostAndFound. By benchmarking Grounding DINO, YOLO-World, MDETR, and OmDet-Turbo with prompt-based inputs and exploring prompt engineering and ensembles, it reveals dataset-specific strengths (e.g., Grounding DINO on RoadObstacle21; YOLO-World on RoadAnomaly21) and consistent limitations such as prompt sensitivity and localization reliability. The findings demonstrate that open-vocabulary detectors hold promise for OOD detection but exhibit substantial gaps before reliable real-world deployment, including multiple predictions per object and struggles with negations in prompts. The work provides a simple prompt-based baseline and points to future directions in robust prompt design, better grounding between language and vision, and improved localization under open-world conditions.

Abstract

Out-of-distribution (OOD) object detection is a critical task focused on detecting objects that originate from a data distribution different from that of the training data. In this study, we investigate to what extent state-of-the-art open-vocabulary object detectors can detect unusual objects in street scenes, which are considered as OOD or rare scenarios with respect to common street scene datasets. Specifically, we evaluate their performance on the OoDIS Benchmark, which extends RoadAnomaly21 and RoadObstacle21 from SegmentMeIfYouCan, as well as LostAndFound, which was recently extended to object level annotations. The objective of our study is to uncover short-comings of contemporary object detectors in challenging real-world, and particularly in open-world scenarios. Our experiments reveal that open vocabulary models are promising for OOD object detection scenarios, however far from perfect. Substantial improvements are required before they can be reliably deployed in real-world applications. We benchmark four state-of-the-art open-vocabulary object detection models on three different datasets. Noteworthily, Grounding DINO achieves the best results on RoadObstacle21 and LostAndFound in our study with an AP of 48.3% and 25.4% respectively. YOLO-World excels on RoadAnomaly21 with an AP of 21.2%.

Figures (7)

• Figure 1: Comparison of OOD object detection using two different models. The left image shows YOLOv8 incorrectly predicting everything as background and missing OOD objects, while the right image shows Grounding DINO detecting both road objects.
• Figure 2: Panels (a) and (b) are prompted with text 'unusual items found on an isolated road'. In figure (a) the model fails to understand the prompt completely, thus predicts objects also outside the road region. In (b) the cars are mistaken to be 'unusual items' by the model in the street scene. In (c) and (d), the model preforms well by accurately detecting the OOD objects given the severe conditions.
• Figure 3: An illustration of multiple predictions with different labels, where the labels assigned the highest confidence are the least precise in the context of the scene.
• Figure 4: Qualitative results on the RoadObstacle21 dataset. Each image featuring different weather and road condition with OOD objects of differing sizes and distances.
• Figure 5: Qualitative results on LostAndFound