Temporally-Grounded Language Generation: A Benchmark for Real-Time Vision-Language Models

TL;DR

This work introduces Temporally-Grounded Language Generation (TGLG), a benchmark for evaluating real-time vision-language models on both semantic accuracy and precise temporal alignment. It formalizes TRACE, a jointly informative metric that balances content quality and timing, and curates SoccerNet (perceptual updating) and HoloAssist (contingency awareness) datasets to test these capabilities. To address real-time constraints, the authors propose Vision-Language Models with Time-Synchronized Interleaving (VLM-TSI), which interleave visual and linguistic tokens along a shared timeline and are trained with standard causal language modeling. Experimental results show VLM-TSI consistently outperforms turn-based baselines like VideoLLM-Online, particularly in reducing overlap and improving start/end timing, though overall performance remains modest and highlights the challenge of truly real-time, temporally grounded generation. The work provides a concrete path forward for developing responsive, temporally aware VLMs with potential applications in assistive robotics, education, and interactive media.

Abstract

Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However, real-time interactive environments impose new demands on VLMs, requiring them to generate utterances that are not only semantically accurate but also precisely timed. We identify two core capabilities necessary for such settings -- $\textit{perceptual updating}$ and $\textit{contingency awareness}$ -- and propose a new benchmark task, $\textbf{Temporally-Grounded Language Generation (TGLG)}$, to evaluate them. TGLG requires models to generate utterances in response to streaming video such that both content and timing align with dynamic visual input. To support this benchmark, we curate evaluation datasets from sports broadcasting and egocentric human interaction domains, and introduce a new metric, $\textbf{TRACE}$, to evaluate TGLG by jointly measuring semantic similarity and temporal alignment. Finally, we present $\textbf{Vision-Language Model with Time-Synchronized Interleaving (VLM-TSI)}$, a model that interleaves visual and linguistic tokens in a time-synchronized manner, enabling real-time language generation without relying on turn-based assumptions. Experimental results show that VLM-TSI significantly outperforms a strong baseline, yet overall performance remains modest -- highlighting the difficulty of TGLG and motivating further research in real-time VLMs. Code and data available $\href{https://github.com/yukw777/tglg}{here}$.

Figures (4)

• Figure 1: Turn-based VLMs fail to operate effectively in real-time environments, because they cannot process new perceptual input while generating responses. $[EOS]$ denotes no generation.
• Figure 2: Sports broadcast datasets like SoccerNet cioppa2022soccernet contain dynamic visual events that require robust perceptual updating. Turn-based models like VideoLLM-Online produce semantically and temporally inaccurate utterances with unrealistic overlaps, while real-time models like VLM-TSI generate semantically aligned and precisely timed utterances without overlaps.
• Figure 3: Egocentric interaction datasets like HoloAssist wang2023holoassist capture complex cooperative interactions that require robust contingency awareness. Turn-based models like VideoLLM-Online may produce temporally aligned utterances, but they struggle to generate useful instructions because they fail to account for the consequences of their prior outputs. In contrast, real-time models like VLM-TSI reason over their past utterances and adapt to the evolving scene, resulting in more context-aware guidance.
• Figure 4: VLM-TSI interleaves vision tokens $v_t$ and text tokens $x_\tau$ in a temporally synchronized manner. For simplicity, each frame $f_t$ is encoded as a single vision token $v_t$.