ENTER: Event Based Interpretable Reasoning for VideoQA

TL;DR

ENTER introduces an event-graph based VideoQA framework that renders videos as structured graphs of events connected by temporal, causal, and hierarchical relations. A large language model generates modular Python code to reason over the graph, enabling transparent and interpretable decision-making, while an iterative hierarchical update mechanism densifies the graph and integrates multimodal information as needed. The approach achieves state-of-the-art or competitive results on NExT-QA, IntentQA, and EgoSchema, with ablations confirming the value of denser captions/graphs and multimodal edges for robustness. By making the reasoning process explicit and debuggable, ENTER offers a practical balance between interpretability and context-rich visual reasoning for complex, long-range video questions.

Abstract

In this paper, we present ENTER, an interpretable Video Question Answering (VideoQA) system based on event graphs. Event graphs convert videos into graphical representations, where video events form the nodes and event-event relationships (temporal/causal/hierarchical) form the edges. This structured representation offers many benefits: 1) Interpretable VideoQA via generated code that parses event-graph; 2) Incorporation of contextual visual information in the reasoning process (code generation) via event graphs; 3) Robust VideoQA via Hierarchical Iterative Update of the event graphs. Existing interpretable VideoQA systems are often top-down, disregarding low-level visual information in the reasoning plan generation, and are brittle. While bottom-up approaches produce responses from visual data, they lack interpretability. Experimental results on NExT-QA, IntentQA, and EgoSchema demonstrate that not only does our method outperform existing top-down approaches while obtaining competitive performance against bottom-up approaches, but more importantly, offers superior interpretability and explainability in the reasoning process.

Figures (9)

• Figure 1: Comparison of our proposed method with existing approaches for VideoQA. Bottom-up approaches directly process visual data but their reasoning is not transparent. In contrast, Top-down methods are interpretable but fail to use visual information in the plan or code generation piece. In contrast, our intermediate multimodal graph-based method combines the strengths of both: interpretability + use of visual information for event graph generation, making our predictions more transparent and context-aware.
• Figure 2: Overall pipeline of our method. Video is first converted to event-graph via captioning as the intermediate step. Next, python code is generated, contextualized by event-graph, to parse the graph. Further, any missing information in graph is added via iterative hierarchical update. Bottom-left flow chart depicts the flow of our method, along with the relative cost of each operation.
• Figure 3: Qualitative examples of ENTER on NExT-QA. The output of our method shows interpretability while maintaining correctness.
• Figure 4: ENTER's interpretability allows the diagnosis and classification of errors into three major types: insufficient knowledge, inconsistent referencing, and missed information. Here, we depict insufficient knowledge and missing information error type.
• Figure 5: Qualitative examples of ENTER on NExT-QA, demonstrating interpretable reasoning alongside correct predictions.