Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models

TL;DR

This paper targets real-time video understanding with Multimodal Large Language Models (MLLMs) by identifying global positional continuity as the bottleneck to input–output parallelism. It introduces three continuity-breaking positional encoding strategies—Overlapped Streaming Position Encoding (OSPE), Group-Decoupled Position Encoding (GDPE), and Gap-Isolated Position Encoding (GIPE)—to enable simultaneous perception and generation without modifying model architecture. Empirical results show that GDPE offers the best balance between accuracy, fluency, and robustness across streaming VD and VQA tasks, while maintaining competitive performance with offline baselines; GIPE and OSPE offer complementary trade-offs. A theoretical analysis demonstrates that parallel streaming can achieve up to $2\times$ acceleration under balanced workloads, providing a principled, plug-and-play pathway toward true real-time MLLM inference, with code released at the provided URL.

Abstract

Multimodal Large Language Models (MLLMs) have achieved strong performance across many tasks, yet most systems remain limited to offline inference, requiring complete inputs before generating outputs. Recent streaming methods reduce latency by interleaving perception and generation, but still enforce a sequential perception-generation cycle, limiting real-time interaction. In this work, we target a fundamental bottleneck that arises when extending MLLMs to real-time video understanding: the global positional continuity constraint imposed by standard positional encoding schemes. While natural in offline inference, this constraint tightly couples perception and generation, preventing effective input-output parallelism. To address this limitation, we propose a parallel streaming framework that relaxes positional continuity through three designs: Overlapped, Group-Decoupled, and Gap-Isolated. These designs enable simultaneous perception and generation, allowing the model to process incoming inputs while producing responses in real time. Extensive experiments reveal that Group-Decoupled achieves the best efficiency-performance balance, maintaining high fluency and accuracy while significantly reducing latency. We further show that the proposed framework yields up to 2x acceleration under balanced perception-generation workloads, establishing a principled pathway toward speak-while-watching real-time systems. We make all our code publicly available: https://github.com/EIT-NLP/Speak-While-Watching.

Figures (9)

• Figure 1: Illustration of different paradigms for video description and positional encoding.The first row shows the offline paradigm, where the model generates the description after observing the entire video, leading to temporal misalignment between narration and visual sequence. The second row presents the interleaved streaming paradigm, which alternates between perception and generation, providing more immediate responses and better temporal coherence, but still suffers from the continuity constraint of positional encoding that prevents full parallelism. The third row illustrates our proposed parallel streaming paradigm, which breaks this continuity, enabling simultaneous perception and generation for true real-time video understanding. The positional IDs serves as a conceptual reference, showing how relaxing positional continuity enables parallel processing between input and output. Please zoom in for a clearer view of details.
• Figure 2: Comparison of different position encoding strategies, where $V_i$ represents the video token sequence from the $i$-th input clip, and $A_i$ represents the corresponding textual output token sequence. Arrows denote the source dependency for the first generated token of each textual output segment. (a) Previous Position Encoding: assigns consecutive positions strictly following the interleaved video-text streaming order; (b) Overlapped Streaming Position Encoding (OSPE): enables video-text streaming parallelism by allowing temporal overlap between encoding and decoding; (c) Group-Decoupled Position Encoding (GDPE): divides video and text into independent groups that maintain intra-group continuity while being inter-group decoupled; (d) Gap-Isolated Position Encoding (GIPE): introduces a fixed gap between groups to fully isolate their index spaces and further reduce cross-modal interference.
• Figure 3: Causal mask visualization. (left) Casual mask for previous video-text interleaved streaming paradigm. (right) Casual mask for parallel streaming paradigm.
• Figure 4: Example of the generated caption by Interleave under random scheduling. Duplicated, fragmented, and grammatically broken segments are highlighted in yellow, while correctly recognized key objects and actions are highlighted in red.
• Figure 5: LLM-as-Judge fluency under scheduling disturbance. The two colors correspond to: (1) trained and evaluated under fixed wait-$K=3$, and (2) trained with wait-$K=3$ but evaluated under random scheduling (disturbance setting).