Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning

TL;DR

This work proposes an agent-based approach to enhance both the efficiency and effectiveness of long-form video understanding by utilizing large language models (LLMs) and their tool-harnessing ability, which leverages the reasoning capabilities of LLMs to process only the most relevant frames in real-time.

Abstract

Understanding long-form video content presents significant challenges due to its temporal complexity and the substantial computational resources required. In this work, we propose an agent-based approach to enhance both the efficiency and effectiveness of long-form video understanding by utilizing large language models (LLMs) and their tool-harnessing ability. A key aspect of our method is query-adaptive frame sampling, which leverages the reasoning capabilities of LLMs to process only the most relevant frames in real-time, and addresses an important limitation of existing methods which typically involve sampling redundant or irrelevant frames. To enhance the reasoning abilities of our video-understanding agent, we leverage the self-reflective capabilities of LLMs to provide verbal reinforcement to the agent, which leads to improved performance while minimizing the number of frames accessed. We evaluate our method across several video understanding benchmarks and demonstrate that not only it enhances state-of-the-art performance but also improves efficiency by reducing the number of frames sampled.

Figures (10)

• Figure 1: Comparison of methods: Our proposed method (c) is query adaptive, dynamically selecting frames based on query and video input to construct a responsive memory. In contrast, previous methods, including (a) Naïve agents and (b) Agents with pre-constructed memory, do not adapt to specific queries or utilize memory dynamically. We demonstrate that dynamically sampling frames have advantage over different set of benchmarks.
• Figure 2: Overall Framework. The video metadata and question are provided to the agent to generate policy, which includes analyzing the question type and determining the task-solving strategy, including the sampling strategy. Planner/Tool Executor, based on the ReAct-style reasoning, generates thought processes, actions, and action inputs, and receives observations from the tools. During this stage, the sampler may suggest improved frames. After formulating the final answer, the evaluator and refiner are applied. The final result is then stored in long-term memory.
• Figure 3: Example of Ego4d NLQ Instance. The User Prompt includes the video's metadata and the question for the Agent to address. (1) Policy Generation: the agent generates an analysis of the question and a sampling strategy (2) Thoughts, Actions and Observation: The agent formulates a Thought based on current state, executes an Action $\mathcal{A}$, with Action Input, and uses tools to obtain an Observation $\mathcal{O}$. This process iterates until the agent comes up with the final answer. (3) Evaluation: the Final Answer is assessed. (4) Refinement: The trajectory $\mathcal{T}$ is refined, and the results are stored in Long-term Memory $\mathcal{M}_{\text{Long}}$.
• Figure 4: Egoschema Results. The number of frames accessed and Accuracy.
• Figure 5: Frame accessed ratio based on textual cues from NextQA benchmark. Darker color corresponds to the higher ratio of access.