Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models

TL;DR

This work addresses the knowledge gap in video understanding by integrating a Vision Foundation Model with a Large Language Model to enable high-level event reasoning and future prediction. It introduces a Q-Former–inspired Vision-Language Fusion Core that distills rich spatiotemporal and object-centric visual information into a compact, language-aligned representation, which grounds the LLM’s reasoning in visual evidence. A two-stage training strategy—alignment pre-training on large video-caption data followed by instruction-based fine-tuning with synthetic causal data—yields state-of-the-art performance on diverse video reasoning benchmarks and strong zero-shot generalization. The approach demonstrates substantial practical impact for embodied AI and robotics, enabling more capable cognitive perception systems that can reason about causes and anticipate future events while highlighting areas for efficiency and grounding improvements.

Abstract

Current video understanding models excel at recognizing "what" is happening but fall short in high-level cognitive tasks like causal reasoning and future prediction, a limitation rooted in their lack of commonsense world knowledge. To bridge this cognitive gap, we propose a novel framework that synergistically fuses a powerful Vision Foundation Model (VFM) for deep visual perception with a Large Language Model (LLM) serving as a knowledge-driven reasoning core. Our key technical innovation is a sophisticated fusion module, inspired by the Q-Former architecture, which distills complex spatiotemporal and object-centric visual features into a concise, language-aligned representation. This enables the LLM to effectively ground its inferential processes in direct visual evidence. The model is trained via a two-stage strategy, beginning with large-scale alignment pre-training on video-text data, followed by targeted instruction fine-tuning on a curated dataset designed to elicit advanced reasoning and prediction skills. Extensive experiments demonstrate that our model achieves state-of-the-art performance on multiple challenging benchmarks. Notably, it exhibits remarkable zero-shot generalization to unseen reasoning tasks, and our in-depth ablation studies validate the critical contribution of each architectural component. This work pushes the boundary of machine perception from simple recognition towards genuine cognitive understanding, paving the way for more intelligent and capable AI systems in robotics, human-computer interaction, and beyond.

Figures (3)

• Figure 1: The overall architecture of our proposed framework. A raw video is processed by the Visual Perception Backbone to extract spatiotemporal and object-centric tokens. The Vision-Language Fusion Core then distills this visual information into a sequence of fused embeddings. Finally, these embeddings, along with a textual task prompt, are fed into the LLM-based Cognitive Reasoner to generate the final textual response for reasoning or prediction.
• Figure 2: An example of a failure case (factual hallucination). Our model correctly describes the overall procedure but incorrectly introduces an object (a torque wrench) that was not present in the video, likely due to strong prior knowledge from its training data. This highlights the challenge of maintaining perfect visual grounding.
• Figure 3: (a) Example of causal reasoning. Our model offers a detailed chain of cause-and-effect, whereas the baseline only provides a terse description.