Temporal Grounding of Activities using Multimodal Large Language Models
Young Chol Song
TL;DR
The paper tackles temporal grounding by proposing a two-stage pipeline that first uses an image-based multimodal LLM to generate frame-level action descriptions and then a text-based LLM to infer the start-end interval of the target activity. It demonstrates that instruction-tuning a smaller multimodal LLM and leveraging descriptions in the second stage yields improved localization on Charades-STA compared to baseline video-based LLMs. Across extensive experiments, the two-stage approach, especially with GPT-4 Vision and GPT-4, approaches or exceeds the performance of contemporary video-grounding LLMs, though specialized vision models still retain edge under some metrics. The work highlights the value of combining detailed image-driven descriptions with language-only reasoning to enhance temporal activity localization and video understanding more broadly.
Abstract
Temporal grounding of activities, the identification of specific time intervals of actions within a larger event context, is a critical task in video understanding. Recent advancements in multimodal large language models (LLMs) offer new opportunities for enhancing temporal reasoning capabilities. In this paper, we evaluate the effectiveness of combining image-based and text-based large language models (LLMs) in a two-stage approach for temporal activity localization. We demonstrate that our method outperforms existing video-based LLMs. Furthermore, we explore the impact of instruction-tuning on a smaller multimodal LLM, showing that refining its ability to process action queries leads to more expressive and informative outputs, thereby enhancing its performance in identifying specific time intervals of activities. Our experimental results on the Charades-STA dataset highlight the potential of this approach in advancing the field of temporal activity localization and video understanding.