VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs

TL;DR

VideoScaffold tackles the challenge of streaming video understanding with multimodal large language models by introducing Elastic-scale Event Segmentation (EES) to adapt event granularity in real time and Hierarchical Event Consolidation (HEC) to generate multi-level, semantically rich representations. EES uses next-frame prediction across multiple hierarchy levels and triggers boundaries based on the prediction error $\mathcal{E}^{(l)}_{t+1}=1-\cos(\hat{\boldsymbol{z}}^{(l)}_{t+1},\boldsymbol{z}^{(l)}_{t+1})>\epsilon^{(l)}$, enabling elastic segmentation that scales with video duration. HEC identifies semantic anchors via essential tokens, then employs cross-attention to fuse intra- and inter-layer context, producing an abstract event embedding $\boldsymbol{e}_{\text{abstract}}$ plus coarse and fine representations for downstream reasoning. The approach achieves state-of-the-art results on both offline and streaming benchmarks, is modular and plug-and-play with existing image-based MLLMs, and demonstrates strong performance with a 7B backbone on long-form tasks and real-time streaming scenarios.

Abstract

Understanding long videos with multimodal large language models (MLLMs) remains challenging due to the heavy redundancy across frames and the need for temporally coherent representations. Existing static strategies, such as sparse sampling, frame compression, and clustering, are optimized for offline settings and often produce fragmented or over-compressed outputs when applied to continuous video streams. We present VideoScaffold, a dynamic representation framework designed for streaming video understanding. It adaptively adjusts event granularity according to video duration while preserving fine-grained visual semantics. VideoScaffold introduces two key components: Elastic-Scale Event Segmentation (EES), which performs prediction-guided segmentation to dynamically refine event boundaries, and Hierarchical Event Consolidation (HEC), which progressively aggregates semantically related segments into multi-level abstractions. Working in concert, EES and HEC enable VideoScaffold to transition smoothly from fine-grained frame understanding to abstract event reasoning as the video stream unfolds. Extensive experiments across both offline and streaming video understanding benchmarks demonstrate that VideoScaffold achieves state-of-the-art performance. The framework is modular and plug-and-play, seamlessly extending existing image-based MLLMs to continuous video comprehension. The code is available at https://github.com/zheng980629/VideoScaffold.

Figures (4)

• Figure 1: Comparative analysis of frame-sampling strategies for video understanding. (a) Uniform sampling is common in offline settings but lacks adaptability to dynamic content.figs/ (b) Cluster-based segmentation groups visually similar frames into events while ignoring temporal order, producing disordered segments. (c) Similarity-based segmentation detects event changes from adjacent-frame similarity, yet fixed thresholds often cause over-fragmentation or incorrect merging. (d) Our Elastic-scale Event Segmentation (EES) continuously refines boundaries and preserves temporal continuity, forming elastic multi-granularity structures that adapt to varying video durations.
• Figure 2: Overview of the proposed VideoScaffold, which comprises two core components: Elastic-Scale Event Segmentation (EES) and Hierarchical Event Consolidation (HEC). As streaming video progresses, a next-frame prediction mechanism continuously refines event boundaries and constructs a hierarchical event structure that adapts to video duration. Upon receiving a user instruction, HEC performs bottom-up summarization along this hierarchy, generating multi-granularity event representations for subsequent video understanding.
• Figure 3: Visualization of the proposed elastic-scale event structure. (a) Layer-wise segmentation results showing how VideoScaffold adaptively refines temporal boundaries and preserves continuity across different video durations (60 and 100 frames). (b) Comparison of intra-event and inter-event similarity on 100 videos (120 frames) from MLVU, highlighting VideoScaffold’s ability to yield semantically cohesive events and distinct inter-event boundaries compared with clustering- or similarity-based methods.
• Figure 4: Ablation on essential frame selection and consolidation.