VCBench: A Streaming Counting Benchmark for Spatial-Temporal State Maintenance in Long Videos

Abstract

Video understanding requires models to continuously track and update world state during playback. While existing benchmarks have advanced video understanding evaluation across multiple dimensions, the observation of how models maintain world state remains insufficient. We propose VCBench, a streaming counting benchmark that repositions counting as a minimal probe for diagnosing world state maintenance capability. We decompose this capability into object counting (tracking currently visible objects vs.\ tracking cumulative unique identities) and event counting (detecting instantaneous actions vs.\ tracking complete activity cycles), forming 8 fine-grained subcategories. VCBench contains 406 videos with frame-by-frame annotations of 10,071 event occurrence moments and object state change moments, generating 1,000 streaming QA pairs with 4,576 query points along timelines. By observing state maintenance trajectories through streaming multi-point queries, we design three complementary metrics to diagnose numerical precision, trajectory consistency, and temporal awareness. Evaluation on mainstream video-language models shows that current models still exhibit significant deficiencies in spatial-temporal state maintenance, particularly struggling with tasks like periodic event counting. VCBench provides a diagnostic framework for measuring and improving state maintenance in video understanding systems.

Figures (6)

• Figure 1: Streaming counting as a probe for spatial-temporal state maintenance. Top: Object counting tracks the current count at each moment (current-state) and cumulative unique identities across time. Bottom: Event counting detects instantaneous actions and span events. Models are queried at multiple timepoints during video playback.
• Figure 2: Taxonomy of spatial-temporal state maintenance. We decompose counting into 8 subcategories across object and event dimensions. Each subcategory requires tracking different types of information: O1 tracks currently visible quantities, O2 tracks cumulative unique identities, E1 counts discrete instantaneous occurrences, and E2 counts complete activity cycles.
• Figure 3: VCBench dataset construction pipeline. Shows the complete process from video source selection, fine-grained annotation generation to streaming query point design. We collect videos from multiple sources including web platforms, existing computer vision datasets, and self-generated simulations, then perform manual annotations and design streaming query points.
• Figure 4: VCBench dataset analysis. Left: Histogram of query point distribution along timelines, showing temporal coverage of streaming queries. Center: Word cloud of counting target semantics, showing diversity of object and event categories. Right: Pie chart of video source distribution, showing scene diversity of the dataset.
• Figure 5: Impact of explicit count annotation on E2-Periodic performance (scaled to 0-100).