Dynamic Scene Understanding from Vision-Language Representations

TL;DR

This paper introduces a unified framework for dynamic scene understanding that exploits frozen vision-language representations to handle both high-level (SiR, HHI) and grounded (HOI, GSR) tasks from a single image. It presents two complementary pathways: structured text prediction for global understanding and attention-based feature augmentation for grounded predictions, with BLIP-2 embeddings often yielding the strongest results. Across four benchmarks, the approach achieves state-of-the-art performance while using relatively few trainable parameters, and analysis demonstrates that modern V&L models encode dynamic scene semantics. The work highlights the value of vision-language pretraining for complex scene understanding and points to future directions in pretraining strategies and grounding enhancements to broaden applicability and impact.

Abstract

Images depicting complex, dynamic scenes are challenging to parse automatically, requiring both high-level comprehension of the overall situation and fine-grained identification of participating entities and their interactions. Current approaches use distinct methods tailored to sub-tasks such as Situation Recognition and detection of Human-Human and Human-Object Interactions. However, recent advances in image understanding have often leveraged web-scale vision-language (V&L) representations to obviate task-specific engineering. In this work, we propose a framework for dynamic scene understanding tasks by leveraging knowledge from modern, frozen V&L representations. By framing these tasks in a generic manner - as predicting and parsing structured text, or by directly concatenating representations to the input of existing models - we achieve state-of-the-art results while using a minimal number of trainable parameters relative to existing approaches. Moreover, our analysis of dynamic knowledge of these representations shows that recent, more powerful representations effectively encode dynamic scene semantics, making this approach newly possible.

Figures (8)

• Figure 1: Given an input image depicting a dynamic scene (left), our framework performs a variety of dynamic scene understanding tasks, such as human-object interactions, human-human and recognition of grounded situations, (A, B, C respectively above). Each of these predicts different entities and relations, possibly grounded in the input image (visualized as bounding boxes on the right). Our generic method contrasts with previous approaches tailored to a single such task.
• Figure 2:
• Figure 3: Grounded Situation Recognition (GSR) qualitative results. Results on the SWiG pratt2020grounded benchmark using our attention feature augmentation method applied to CoFormer cho2022collaborative. The predicted main activity for image is indicated below it, while the corresponding predicted semantic roles (arguments) are displayed in the table, with nouns labeled according to their specific roles within the activity. Bounding boxes for the predicted AGENT are shown in pink and other predicted roles' boxes are shown in green. As demonstrated, our method successfully predicts complex situations, including those involving non-human agents.
• Figure 4: Qualitative results. Results of our framework over several dynamic scene understanding tasks: (a) Human-Human Interaction (HHI), (b) Situation Recognition (SiR), (c) Grounded Situation Recognition (GSR), and (d) Human-Object Interaction Detection (HOI). For further details on these tasks, see Section \ref{['sec:tasks']}
• Figure 5: Situation Recognition (SiR) qualitative comparison. Comparison of results between CoFormer cho2022collaborative and our method (CoFormer+) on the imSitu yatskar2016situationpratt2020grounded test set. We apply our our proposed attention feature augmentation mechanism to a CoFormer backbone. Incorrect predictions are shown in red.