Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding

TL;DR

Deep Video Discovery addresses the problem of understanding ultra-long videos by introducing an autonomous agent that performs adaptive, tool-guided search over a multi-granular video database. The method combines a triad of tools—Global Browse, Clip Search, and Frame Inspect—with an LLM-powered planner in an iterative reasoning loop to gather evidence and answer queries. It achieves state-of-the-art results on LVBench and other long-video benchmarks, supported by comprehensive ablations and analyses of tool contributions and reasoning behavior. This work demonstrates the practicality and impact of agentic search for scalable, long-form video understanding, while acknowledging computational cost as a trade-off and outlining avenues for efficiency improvements.

Abstract

Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. To overcome such limitations, we propose the Deep Video Discovery (DVD) agent to leverage an agentic search strategy over segmented video clips. Unlike previous video agents that rely on predefined workflows applied uniformly across different queries, our approach emphasizes the autonomous and adaptive nature of agents. By providing a set of search-centric tools on multi-granular video database, our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information. We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates our advantage. Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of 74.2%, which substantially surpasses all prior works, and further improves to 76.0% with transcripts. The code has been released at https://github.com/microsoft/DeepVideoDiscovery.

Figures (8)

• Figure 1: Left: Illustration of our Deep Video Discovery agent, which autonomously reasons on user query, iterative use tools to obtain the final answer. Right: Performance comparison on LVBench.
• Figure 2: Deep Video Discovery consists of two stages: 1) Multi-granular Video Database Construction. We extract video information from different levels to enable comprehensive understanding, efficient retrieval, and preservation of original content. 2) Agentic Search and Answer. The agent iteratively reasons on user query and leverage the tailored toolset to gather information to answer.
• Figure 3: Analysis of the behavior of Deep Video Discovery using different reasoning models. We categorize tool-calling behavior into five types. For each type, we report its proportion (Ratio, sector angels), average reasoning steps (Steps, sector radius) and score (Score, dashed lines). A clear correlation emerges among behavior patterns, reasoning depth, and score (see Section \ref{['sec:exp-analysis']} for details).
• Figure 4: Case study of Global Browse Only behavior.
• Figure 5: Case study of Simple Action behavior.