Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

Linli Yao, Yuancheng Wei, Yaojie Zhang, Lei Li, Xinlong Chen, Feifan Song, Ziyue Wang, Kun Ouyang, Yuanxin Liu, Lingpeng Kong, Qi Liu, Pengfei Wan, Kun Gai, Yuanxing Zhang, Xu Sun

TL;DR

OmniDenseCaptioning introduces a dense, time-aware framework for multi-scene audio-visual narration through a six-dimension caption schema and explicit timestamps. The authors present OmniDCBench, a 1,122-video benchmark with comprehensive human annotations, and SodaM, a unified evaluation metric that aligns temporal segmentation with caption completeness using an IoU-based dynamic programming alignment and per-dimension checks. TimeChat-Captioner-7B, trained via SFT and GRPO with task-specific rewards, achieves state-of-the-art performance on OmniDCBench and consistently boosts downstream tasks such as Daily-Omni, World-Sense, and Charades-STA, demonstrating strong omni-modal alignment and temporal grounding. The work provides high-quality synthetic training data (TimeChatCap-42K) and a rigorous evaluation framework, enabling robust progress toward dense, time-aligned omni-video understanding and generation while outlining future improvements for longer contexts and efficiency.

Abstract

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code will be made publicly available at https://github.com/yaolinli/TimeChat-Captioner.

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

TL;DR

OmniDenseCaptioning introduces a dense, time-aware framework for multi-scene audio-visual narration through a six-dimension caption schema and explicit timestamps. The authors present OmniDCBench, a 1,122-video benchmark with comprehensive human annotations, and SodaM, a unified evaluation metric that aligns temporal segmentation with caption completeness using an IoU-based dynamic programming alignment and per-dimension checks. TimeChat-Captioner-7B, trained via SFT and GRPO with task-specific rewards, achieves state-of-the-art performance on OmniDCBench and consistently boosts downstream tasks such as Daily-Omni, World-Sense, and Charades-STA, demonstrating strong omni-modal alignment and temporal grounding. The work provides high-quality synthetic training data (TimeChatCap-42K) and a rigorous evaluation framework, enabling robust progress toward dense, time-aligned omni-video understanding and generation while outlining future improvements for longer contexts and efficiency.

Abstract

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code will be made publicly available at https://github.com/yaolinli/TimeChat-Captioner.
Paper Structure (30 sections, 5 equations, 7 figures, 9 tables)

This paper contains 30 sections, 5 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio-Visual Captioning
  4. Time-Aware Video Captioning
  5. Reinforcement Learning for Video Captioning
  6. OmniDenseCaptioning Task and A New Benchmark
  7. Task Definition
  8. Benchmark Dataset Curation
  9. Evaluation Metric Design
  10. The TimeChat-Captioner Framework
  11. Overall Architecture
  12. Training Data Collection
  13. Training Strategy
  14. SFT Stage
  15. GRPO Stage
  16. ...and 15 more sections

Figures (7)

  • Figure 1: Illustration of the OmniDenseCaptioning task. This paper introduces Omni Dense Captioning task, which generates fine-grained, temporally grounded descriptions for comprehensive audio-visual understanding. The term "dense" reflects two key properties: (1) temporally-dense: continuous scene segmentation with explicit timestamps, and (2) description-dense: structured captions spanning six dimensions: Auiod-Visual Events, Visual Background, Camera State, Shot Editing, Dialogue, and Acoustic cues. These "script-like" descriptions allow readers to imagine the video scene-by-scene, as though reading a cinematic screenplay.
  • Figure 2: Statistics of human-annotated OmniDCBench.(a) Video duration distribution. (b) Caption length distribution with per-dimension. The benchmark features comprehensive annotations averaging 995 words per video. (c) Scene duration distribution (in seconds), compared against MLLM-generated outputs to highlight the granularity gap between human and model segmentations.
  • Figure 3: Overview of TimeChat-Captioner Architecture.(Left) This model leverages Qwen2.5-Omni qwen2d5omni with interleaved audio-visual tokens to generate multi-scene timestamps and six-dimensional captions. (Right) Two-stage training: SFT for task format learning, followed by GRPO with rewards for format, length, timestamp accuracy, and time-aware fine-grained caption quality.
  • Figure 4: Qualitative case analysis. We compare TimeChat-Captioner with Gemini-2.5-Pro and Qwen-3-Omni on a sample from OmniDCBench. Our model achieves fine-grained alignment with the ground truth across all six annotation dimensions: detailed events, visual background, acoustics, dialogue, camera state, and shot editing style. In contrast, Gemini-2.5-Pro gemini exhibits severe hallucination by misidentifying the male driver as a woman, fundamentally distorting the scene semantics. Qwen-3-Omni Qwen3-Omnimisses the main event entirely, describing irrelevant background elements (a doorman in red uniform) while ignoring the central conversation inside the car. These results demonstrate TimeChat-Captioner's superior capability in accurate character recognition, faithful event grounding, and comprehensive multi-dimensional annotation.
  • Figure 5: Statistics of the training dataset TimeChatCap-42K.(a) Video duration distribution; most videos (73.9%) fall within 50-60 seconds. (b) Caption length distribution with per-dimension average word counts; annotations average 877 words per video across six dimensions. (c) Segment duration distribution; the average segment length is 10.04 seconds.
  • ...and 2 more figures