SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models

TL;DR

SOVABench introduces a real-world vehicle-surveillance action retrieval benchmark with inter-pair and intra-pair protocols to evaluate cross-action discrimination and temporal-direction understanding. It also proposes a training-free MLLM-to-Embedding framework that converts MLLM-described visual content into sentence-level embeddings, using two prompting strategies to enhance task relevance. Empirical results show that instruction-conditioned embeddings can outperform CLIP on image-based tasks and provide strong inter-pair performance on SOVABench, though intra-pair discrimination remains challenging due to description-generation limitations. The work advances interpretable, action-focused retrieval in surveillance and highlights directions for improving temporal reasoning and prompt design in multimodal embeddings.

Abstract

Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video retrieval benchmarks focus on scene-level similarity and do not evaluate the action discrimination required in surveillance. To address this gap, we introduce SOVABench (Surveillance Opposite Vehicle Actions Benchmark), a real-world retrieval benchmark built from surveillance footage and centered on vehicle-related actions. SOVABench defines two evaluation protocols (inter-pair and intra-pair) to assess cross-action discrimination and temporal direction understanding. Although action distinctions are generally intuitive for human observers, our experiments show that they remain challenging for state-of-the-art vision and multimodal models. Leveraging the visual reasoning and instruction-following capabilities of Multimodal Large Language Models (MLLMs), we present a training-free framework for producing interpretable embeddings from MLLM-generated descriptions for both images and videos. The framework achieves strong performance on SOVABench as well as on several spatial and counting benchmarks where contrastive Vision-Language Models often fail. The code, annotations, and instructions to construct the benchmark are publicly available.

Figures (8)

• Figure 1: Samples and performance in SOVABench. (a) Illustrative samples of the constructed benchmark of four different actions (close trunk, open vehicle door, start, and turn left), and (b) comparison of methods for the two evaluation protocols in SOVABench. Methods include MLLMs using the MLLM-to-Embedding framework to obtaining embeddings (bold) and contrastive VLMs. For reference, random values are 3.4 mAP and 50.3 Pair-mAP in Inter-pair and Intra-pair protocols respectively.
• Figure 2:
• Figure 3:
• Figure 4:
• Figure 6: Overview of the MLLM-to-Embedding framework. Given an image/video and the textual instruction, an MLLM first generates a descriptive textual response. The output is then split into individual sentences, each encoded using a sentence-similarity text encoder. The similarity between two samples is computed using the maximum pairwise cosine similarity between their sentence embeddings (Equation \ref{['eq:similarity']}). The filmstrip is extracted from SOVABench.