MR. Video: "MapReduce" is the Principle for Long Video Understanding

TL;DR

MR. Video introduces a simple yet effective MapReduce principle for long video understanding, addressing the context-length limits of Vision-Language Models and the scalability challenges of video agents. The framework splits processing into two stages: Captioning (Map: dense, scene-level captions; Reduce: unify character/object names) and Analysis (I: question intention analysis; II: goal-aware analysis with final answer generation). Through extensive experiments on LVBench and related benchmarks, MR. Video achieves strong improvements (>10% relative) over state-of-the-art methods, and ablations confirm the value of consistent naming, intention analysis, and goal-aware reasoning. The work highlights the potential of combining MapReduce with LLM-driven perception to enable scalable, global-context-aware long video understanding, with clear prompts and implementation details provided for replication and future exploration.

Abstract

We propose MR. Video, an agentic long video understanding framework that demonstrates the simple yet effective MapReduce principle for processing long videos: (1) Map: independently and densely perceiving short video clips, and (2) Reduce: jointly aggregating information from all clips. Compared with sequence-to-sequence vision-language models (VLMs), MR. Video performs detailed short video perception without being limited by context length. Compared with existing video agents that typically rely on sequential key segment selection, the Map operation enables simpler and more scalable sequence parallel perception of short video segments. Its Reduce step allows for more comprehensive context aggregation and reasoning, surpassing explicit key segment retrieval. This MapReduce principle is applicable to both VLMs and video agents, and we use LLM agents to validate its effectiveness. In practice, MR. Video employs two MapReduce stages: (A) Captioning: generating captions for short video clips (map), then standardizing repeated characters and objects into shared names (reduce); (B) Analysis: for each user question, analyzing relevant information from individual short videos (map), and integrating them into a final answer (reduce). MR. Video achieves over 10% accuracy improvement on the challenging LVBench compared to state-of-the-art VLMs and video agents. Code is available at: https://github.com/ziqipang/MR-Video

Figures (9)

• Figure 1: MapReduce Principle. Long video understanding requires both global comprehension and detailed perception, as in the motivating example. For such needs, (a) sequence-to-sequence VLMs and (b) video agents are sub-optimal in terms of context lengths, sequential parallelization, and global context information. (c) Instead, we explore and develop a simple "MapReduce" principle via MR. Video and (d) overcome such challenges.
• Figure 2: Overview. MR. Video validates the effectiveness of "MapReduce" principle with an LLM agent framework. We demonstrate two distinct types of questions for visual details and reasoning. (a)Captioning (Sec. \ref{['sec:captioning']}) first generates detailed captions of individual scenes (Map) and then enhances consistency by merging repeated characters/objects for the scenes (Reduce). (b)Question Intention Analysis (Sec. \ref{['sec:analysis_1']}) investigates if a video segment contributes useful information (Map) and then forms a unified analysis at the video level (Reduce). (c)Goal-Aware Analysis (Sec. \ref{['sec:analysis_2']}) delves deep into detailed perception and reasoning with available contexts, guided by the intention analysis (Map), then unifies them into a final answer (Reduce). For clarity, MR. Video's intermediate texts are simplified.
• Figure 3: Key Characters/Objects in Captioning. (a) The "map" step extracts the salient characters/objects along with a description, which is useful for frames with multiple characters (the 3rd frame). (b) Then, the "reduce" step uses VLM to associate the repeated characters, enhancing the consistency of captions.
• Figure 4: Question Intention Analysis. Long video questions generally require the model to recover certain information, e.g., the meaning of "protagonist", what a "utility room" looks like, and confounding relevant segments, as in (a). This motivates MR. Video's explicitly understanding the question's intentions by reasoning both the video contents and questions.
• Figure 5: Customized Queries for Perception. With this question requiring detailed visual perception, MR. Video proposes objective-aware queries for the VLMs, confirming the criteria.