Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering

TL;DR

This work tackles verb ambiguity in visual activity recognition by introducing a two-step vision-language clustering framework that builds visually grounded verb sense clusters. By generating <image,verb> pairs with multimodal LLMs and clustering embeddings, the method yields approximately four sense clusters per image, each containing about two synonyms, enabling a cluster-based evaluation that better matches human judgments than exact-match metrics. The approach demonstrates improved alignment and robustness across supervised and multimodal models, while revealing limitations of CLIP-based verb scoring for action-centric tasks. The framework is designed to be adaptable to other datasets and provides a principled, scalable way to evaluate visual activity understanding under linguistic variability.

Abstract

Evaluating visual activity recognition systems is challenging due to inherent ambiguities in verb semantics and image interpretation. When describing actions in images, synonymous verbs can refer to the same event (e.g., brushing vs. grooming), while different perspectives can lead to equally valid but distinct verb choices (e.g., piloting vs. operating). Standard exact-match evaluation, which relies on a single gold answer, fails to capture these ambiguities, resulting in an incomplete assessment of model performance. To address this, we propose a vision-language clustering framework that constructs verb sense clusters, providing a more robust evaluation. Our analysis of the imSitu dataset shows that each image maps to around four sense clusters, with each cluster representing a distinct perspective of the image. We evaluate multiple activity recognition models and compare our cluster-based evaluation with standard evaluation methods. Additionally, our human alignment analysis suggests that the cluster-based evaluation better aligns with human judgments, offering a more nuanced assessment of model performance.

Figures (5)

• Figure 1: Accuracy score based on the exact match against gold labeling does not consider the ambiguity.
• Figure 2: Overview of our two-step clustering framework. Step 1 groups <image,verb> pairs with the same verb into fine-grained sense clusters. Step 2 merges these across verbs to capture shared semantic meanings.
• Figure 3: Qualitative examples of our clustering output. Each mini-panel shows: the imSitu gold verb (medal icon), Llama-3.2-90B predictions in bold, and additional verbs grouped in the same sense cluster. Different background colors indicate distinct clusters associated with the image.
• Figure 4: Top-1 accuracy of different models based on our clustering results. The dashed baselines represent the accuracy of the models when we use only the Llama-3.2-90B (closed) outputs as target verbs (no clustering), i.e., model's prediction verb is considered correct when it matches any of the targets.
• Figure 5: Error analysis of our clustering pipeline. Inference predictions are shown in bold, and different clusters for the same image are distinguished by background colors.