Streaming Video Instruction Tuning

TL;DR

This work presents Streamo, a real-time streaming video LLM that unifies perception and decision-making for interactive, continuous video streams. It introduces Streamo-Instruct-465K, a large-scale, unified instruction-tuning dataset with standardized temporal annotations across real-time narration, captioning, grounding, and time-sensitive QA, enabling end-to-end online adaptation of offline video models. A three-state decision mechanism (Silence, Standby, Response) is embedded directly in the model to achieve efficient one-pass inference, with a focal-loss-based training objective to address severe class imbalance. Empirical results on online, offline, and multi-task benchmarks show that Streamo outperforms prior online models and preserves offline perceptual capabilities, while Streamo-Bench provides a comprehensive evaluation of instruction-following in streaming contexts. The work advances unified, real-time video understanding and offers a practical pathway to deploying general-purpose streaming video assistants.

Abstract

We present Streamo, a real-time streaming video LLM that serves as a general-purpose interactive assistant. Unlike existing online video models that focus narrowly on question answering or captioning, Streamo performs a broad spectrum of streaming video tasks, including real-time narration, action understanding, event captioning, temporal event grounding, and time-sensitive question answering. To develop such versatility, we construct Streamo-Instruct-465K, a large-scale instruction-following dataset tailored for streaming video understanding. The dataset covers diverse temporal contexts and multi-task supervision, enabling unified training across heterogeneous streaming tasks. After training end-to-end on the instruction-following dataset through a streamlined pipeline, Streamo exhibits strong temporal reasoning, responsive interaction, and broad generalization across a variety of streaming benchmarks. Extensive experiments show that Streamo bridges the gap between offline video perception models and real-time multimodal assistants, making a step toward unified, intelligent video understanding in continuous video streams.

Figures (7)

• Figure 1: An example of multi-task annotation in Streamo-Instruct-465K. Each task is carefully labeled with the corresponding response time boundaries and content, following established annotation standards. The same video is annotated with multiple distinct tasks. The video shown in this example is sourced from ActivityNet caba2015activitynet.
• Figure 2: The format of a multi-turn dialogue.
• Figure 3: Streamo's architecture. Streaming video data is organized into an interleaved, multi-turn dialogue structure that directly integrates a response-state token into the data sequence, enabling end-to-end parallel training.
• Figure 4: Dataset distribution overview. Left: task distribution; Right: video duration distribution.
• Figure 5: Streamo-Bench example illustrating multi-task instruction-following evaluation. The video shown in this example is sourced from COIN tang2019coin.