ELSA: Evaluating Localization of Social Activities in Urban Streets using Open-Vocabulary Detection

TL;DR

This work introduces ELSA, a benchmark and evaluation framework for open-vocabulary detection of social activities in urban street images, featuring multi-label annotations across four label categories and 830 natural-language prompts. It addresses semantic understanding and localization with two novel tools: the context-aware N-LSE confidence score and the Dynamic Box Aggregation (DBA) algorithm, which together improve reliability over traditional max-logit scoring and NMS-based methods. Empirical results across state-of-the-art models show that N-LSE reduces false positives and that DBA mitigates inflated AP while recovering true positives, though performance remains lower on complex, real-world scenes. The dataset, evaluation toolkit, and insights aim to guide future improvements in OVD for social-activity localization in dynamic urban environments.

Abstract

Existing Open Vocabulary Detection (OVD) models exhibit a number of challenges. They often struggle with semantic consistency across diverse inputs, and are often sensitive to slight variations in input phrasing, leading to inconsistent performance. The calibration of their predictive confidence, especially in complex multi-label scenarios, remains suboptimal, frequently resulting in overconfident predictions that do not accurately reflect their context understanding. To understand these limitations, multi-label detection benchmarks are needed. A particularly challenging domain for such benchmarking is social activities. Due to the lack of multi-label benchmarks for social interactions, in this work we present ELSA: Evaluating Localization of Social Activities. ELSA draws on theoretical frameworks in urban sociology and design and uses in-the-wild street-level imagery, where the size of groups and the types of activities vary significantly. ELSA includes more than 900 manually annotated images with more than 4,300 multi-labeled bounding boxes for individual and group activities. We introduce a novel confidence score computation method NLSE and a novel Dynamic Box Aggregation (DBA) algorithm to assess semantic consistency in overlapping predictions. We report our results on the widely-used SOTA models Grounding DINO, Detic, OWL, and MDETR. Our evaluation protocol considers semantic stability and localization accuracy and further exposes the limitations of existing approaches.

Figures (7)

• Figure 1: We present ELSA: Evaluating Localization of Social Activities—a novel benchmark dataset and evaluation framework for assessing open-vocabulary detection (OVD) models in recognizing and localizing social interactions on urban streets from still images. ELSA includes a multi-label annotation scheme spanning four categories: Condition, State, Activity, and Other. It features natural language prompts derived from these labels, along with synonymous variations to rigorously test models' semantic comprehension. Our N-LSE context-aware confidence score surpasses max-logit scoring, yielding more realistic confidence scores and effectively reducing false positives. Our DBA algorithm dynamically groups overlapping predictions, ensuring semantic coherence and recovering correct predictions that otherwise would be missed by class-agnostic NMS.
• Figure 2: Examples of individual annotations extracted from larger images in the benchmark. Each bounding box is accompanied by a set of labels, a base natural language prompt, and a series of synonyms, two of which are shown here.
• Figure 3: Overview of label distribution and combinations in ELSA, showing the 15 most frequently occurring label combinations. Connected dots represent label combinations, with frequencies plotted in the bar charts above each combination. For example, the "walking alone" combination appears 1,363 times, while "standing alone" appears 713 times.
• Figure 4: Using the Grounding DINO model with Swin-T backbone and Max-logit scoring to run variations of the same prompt with different states.
• Figure 5: Comparison of average score of the five most frequent prompts computed using the Max-logit and N-LSE (ours). The plot shows how Max-Logit scores may be artificially inflated.