Semantic Event Graphs for Long-Form Video Question Answering

TL;DR

This work tackles long-form video question answering by introducing Semantic Event Graphs (SEG), a symbolic, memory-efficient interface that converts continuous footage into START/END human–object interactions organized as a Temporal Scene Graph (TSG). A four-stage pipeline—object detection/tracking with YOLOv11, proximity-based event extraction, TSG construction, and query-aware pruning—produces a compact narrative that feeds Gemini 2.5 Flash for QA. Results on five long YouTube videos show SEG achieves 65.0% accuracy using only 3,466 tokens, versus 62.5% with 40,390 tokens for full-context prompting, yielding a token reduction of about 91.4% while maintaining comparable performance. This demonstrates that symbolic temporal graphs can serve as an effective, scalable memory layer for long-horizon video reasoning, enabling token- and cost-efficient QA without sacrificing long-range reasoning capabilities.

Abstract

Long-form video question answering remains challenging for modern vision-language models, which struggle to reason over hour-scale footage without exceeding practical token and compute budgets. Existing systems typically downsample frames or feed dense visual embeddings to large-context language models, trading off temporal coverage against cost. We propose Semantic Event Graphs (SEG), a lightweight symbolic interface between video and language that replaces raw frames with compact temporal interaction logs. Our pipeline detects and tracks objects with YOLOv11, converts proximity patterns into START/END human-object events, and organizes them into a Temporal Scene Graph (TSG). At inference time, a query-aware pruning module identifies anchor entities and lexically relevant events, returning only a small subgraph which is verbalized and passed to Gemini 2.5 Flash for answer generation. On five YouTube videos (300-500 interactions each) and 120 automatically generated long-horizon questions, SEG achieves 65.0% accuracy using only 3.47k tokens per query, closely matching a full-log baseline (62.5% at 40.39k tokens) while reducing token usage by 91.4%. A short-context baseline restricted to the last 30 seconds collapses to 2.5% accuracy, underscoring the need for explicit temporal memory. These results show that symbolic temporal graphs can serve as an effective, plug-and-play memory layer for off-the-shelf vision-language models, preserving long-range reasoning ability while making long-form video question answering substantially more token- and cost-efficient. Code, logs, and event-extraction tools will be released for reproducibility.

Figures (3)

• Figure 1: Example Temporal Scene Graph before (left) and after (right) query-aware pruning for the query "laptop". The full video produces a dense temporal graph over many entities; SEG prunes this to a small subgraph containing only the people and objects needed to answer the question.
• Figure 2: Method pipeline overview: raw video is processed by YOLOv11 detection and tracking, converted into START/END interaction events, assembled into a Temporal Scene Graph (TSG), pruned according to the query, and finally passed as a compact narrative to Gemini 2.5 Flash for question answering.
• Figure 3: Accuracy vs Token Usage for all methods. TSG achieves near-optimal accuracy at drastically lower token cost.