SiLVR: A Simple Language-based Video Reasoning Framework

TL;DR

SiLVR tackles the challenge of complex video-language reasoning by separating perception from reasoning in a simple, training-free two-stage framework. It converts videos into language-based descriptions via dense clip captions and ASR transcripts, then reasons over these descriptions with a strong LLM, aided by Adaptive Context Reduction to handle long input sequences. The approach achieves state-of-the-art results on multiple long-form and grounding video QA benchmarks, and demonstrates notable knowledge acquisition and temporally grounded QA capabilities without video-specific training. Findings indicate that leveraging powerful reasoning LLMs to fuse multisensory information can yield strong video reasoning performance with a modular, plug-and-play design, offering a strong baseline for future multimodal reasoning work.

Abstract

Recent advances in test-time optimization have led to remarkable reasoning capabilities in Large Language Models (LLMs), enabling them to solve highly complex problems in math and coding. However, the reasoning capabilities of multimodal LLMs (MLLMs) still significantly lag, especially for complex video-language tasks. To address this issue, we present SILVR, a Simple Language-based Video Reasoning framework that decomposes complex video understanding into two stages. In the first stage, SILVR transforms raw video into language-based representations using multisensory inputs, such as short clip captions and audio/speech subtitles. In the second stage, language descriptions are fed into a powerful reasoning LLM to solve complex video-language understanding tasks. To handle long-context multisensory inputs, we use an Adaptive Context Reduction scheme, which dynamically determines the temporal granularity with which to sample the tokens. Our simple, modular, and training-free video reasoning framework achieves the best-reported results on Video-MME (long), Video-MMMU (comprehension), Video-MMLU, CGBench, and EgoLife. Furthermore, our empirical study focused on video reasoning capabilities shows that, despite not being explicitly trained on video, strong reasoning LLMs can effectively aggregate multisensory input information from video, speech, and audio for complex temporal, causal, long-context, and knowledge acquisition reasoning tasks in video. More details can be found at https://sites.google.com/cs.unc.edu/silvr.

Figures (9)

• Figure 1: Strong reasoning capabilities of SiLVR on complex video QA tasks. SiLVR leverages recent advances in reasoning LLMs for complex video QA tasks. SiLVR achieves better performance than strong proprietary non-reasoning models (i.e., GPT-4o and Gemini-1.5) on benchmarks like VideoMME (long), Video-MMMU (Comprehension), Video-MMLU, and EgoLife, which include temporal, causal, long-context, and external knowledge reasoning tasks. The example reasoning trace shows SiLVR’s capability to perform self-correction, in which it successfully identifies that shells are decorative rather than functional.
• Figure 2: Method overview. SiLVR is a simple two-stage language-based video reasoning framework. Top: The video is segmented into short clips and paired with speech. A clip captioner processes each segment to generate visual descriptions. The speech is transcribed using ASR. Bottom: A reasoning LLM takes the question, transcribed speech, and dense visual descriptions compressed by Adaptive Context Reduction to perform complex video reasoning. In the shown example, SiLVR infers the correct order by integrating information across both visual and speech modalities. The model correctly identifies the sequence through reasoning and eliminating incorrect options.
• Figure 3: Performance breakdown across different question categories. Using DeepSeek-R1 as an LLM yields larger gains on reasoning questions (+11.1%) than general perception questions (+4.9%). Full category names are in the supplementary materials (Table \ref{['tab:videomme_breakdown_supp']}).
• Figure 4: Example 1 of SiLVR's reasoning trace. The question asks about the size of the back touchscreen in the car shown in the video. The visual captioning module of SiLVR fails to capture the details about the touchscreen, which appears briefly for only about one second. However, by identifying the vehicle type and leveraging external knowledge from the LLM, SiLVR infers the correct answer.
• Figure 5: Results on knowledge acquisition and temporally grounded QA tasks. SiLVR achieves the highest $\Delta_\text{knowledge}$ on VideoMMMU and the best mIoU on CGBench.