A Challenge to Build Neuro-Symbolic Video Agents

TL;DR

The paper addresses the need for proactive, reliable video agents capable of temporal reasoning and real-world interaction. It advocates a neuro-symbolic approach that combines deep learning for perception with temporal-logic-based reasoning to enable structured, explainable decision-making. Key contributions include articulating a formal challenge, introducing the TLV dataset concept, and proposing NeuS-V and NSVS-TL as benchmarks for video search and generation with temporal fidelity, along with metrics for event accuracy and tool invocation. The work highlights the practical impact of trustworthy video agents in domains like home security and autonomous systems, and calls for standardized benchmarks and multi-modal, multi-agent research to realize robust, high-integrity video understanding and action.

Abstract

Modern video understanding systems excel at tasks such as scene classification, object detection, and short video retrieval. However, as video analysis becomes increasingly central to real-world applications, there is a growing need for proactive video agents for the systems that not only interpret video streams but also reason about events and take informed actions. A key obstacle in this direction is temporal reasoning: while deep learning models have made remarkable progress in recognizing patterns within individual frames or short clips, they struggle to understand the sequencing and dependencies of events over time, which is critical for action-driven decision-making. Addressing this limitation demands moving beyond conventional deep learning approaches. We posit that tackling this challenge requires a neuro-symbolic perspective, where video queries are decomposed into atomic events, structured into coherent sequences, and validated against temporal constraints. Such an approach can enhance interpretability, enable structured reasoning, and provide stronger guarantees on system behavior, all key properties for advancing trustworthy video agents. To this end, we present a grand challenge to the research community: developing the next generation of intelligent video agents that integrate three core capabilities: (1) autonomous video search and analysis, (2) seamless real-world interaction, and (3) advanced content generation. By addressing these pillars, we can transition from passive perception to intelligent video agents that reason, predict, and act, pushing the boundaries of video understanding.

Figures (5)

• Figure 1: An Efficient Neuro-Symbolic Approach to Video Agents. We argue for a neuro-symbolic approach to develop video agents that combines the per-frame or short-horizon reasoning capabilities of neural perception models with the long-term reasoning abilities of symbolic frameworks such as temporal logic tools. Here, we show one such example from a home security system, where the agent is required to identify the presence of a delivery person and send the required notifications.
• Figure 2: NSVS-TL and NeuS-V System Diagrams. In (a), when given video feed from a security system, NSVS-TL demonstrates its capability in identifying exactly when a delivery driver walks up the stairs and drops a package off. Similarly, in (b), a video generated by a foundation model describing a delivery scenario is evaluated for temporal fidelity through NeuS-V.
• Figure 3: Foundation models struggle to perform video search and generation with increasing complexity of user queries. However, neuro-symbolic approaches (NSVS-TL) effectively decouple spatial and temporal reasoning using perception modules for spatial reasoning and temporal logic (TL) to model long-term temporal dependencies. As a result, NSVS-TL outperforms foundation models in complex video search tasks (a). Similarly, text-to-video models, like Pika, fail to maintain temporal consistency as scenario complexity increases (b).
• Figure 4: TLV Dataset Specification-Video Pairing. An excerpt of the TLV dataset is shown here, demonstrating the efficacy of the TLV dataset in showing the relationships between videos and TL sequences. This figure was taken with permission from choi2024towards.
• Figure 5: What is the Correct Action for the Agent? At a first glance, both agents summarize the video nearly identically. However, upon closer inspection, Agent 1, although more vague, correctly identifies the parked delivery truck and notifies the homeowner. In contrast, Agent 2, while more detailed, hallucinates a right turn on an intersection, leading to no action.