TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability

TL;DR

TimeMarker tackles the challenge of precise temporal localization in video-LLMs and variable-length videos by introducing Temporal Separator Tokens and an AnyLength sampling/merging mechanism. It builds on a LLaVA-based architecture with a cross-modal projector and a large language model, trained in three stages with image-text and temporally enriched video data, achieving state-of-the-art results on short and long video benchmarks and temporal grounding tasks. The approach enables explicit timestamp grounding, scalable processing, and robust temporal reasoning, with strong zero-shot grounding performance. The work offers a practical path toward versatile video-LLMs capable of precise moment localization across diverse video lengths.

Abstract

Rapid development of large language models (LLMs) has significantly advanced multimodal large language models (LMMs), particularly in vision-language tasks. However, existing video-language models often overlook precise temporal localization and struggle with videos of varying lengths. We introduce TimeMarker, a versatile Video-LLM designed for high-quality dialogue based on video content, emphasizing temporal localization. TimeMarker integrates Temporal Separator Tokens to enhance temporal awareness, accurately marking specific moments within videos. It employs the AnyLength mechanism for dynamic frame sampling and adaptive token merging, enabling effective handling of both short and long videos. Additionally, TimeMarker utilizes diverse datasets, including further transformed temporal-related video QA datasets, to bolster its temporal understanding capabilities. Image and interleaved data are also employed to further enhance the model's semantic perception ability. Evaluations demonstrate that TimeMarker achieves state-of-the-art performance across multiple benchmarks, excelling in both short and long video categories. Our project page is at \url{https://github.com/TimeMarker-LLM/TimeMarker/}.

Figures (6)

• Figure 1: The motivation of our proposed TimeMarker. TimeMarker interleaves textual temporal separator tokens with video frame tokens, explicitly encoding the absolute temporal positions of video frames.
• Figure 2: TimeMarker attains leading performance across a range of comprehensive video understanding benchmarks. In the figure, we only list some 7B/8B models and highlight the specific scores of TimeMarker.
• Figure 3: The overview of our TimeMarker model. TimeMarker builds on the LLaVA architecture, using a Vision Encoder and cross-modality Projector to integrate visual and textual tokens for the LLM. We introduce an AnyLength mechanism with an Adaptive Token Merge module to handle varying video lengths, and Temporal Separator Tokens Integration to encode temporal positions, enabling the LLM to perceive specific timestamps.
• Figure 4: The training data distribution and detailed list used in the PT2 and SFT stages of TimeMarker. In addition to video data, we utilize other multimodal data to assist in training TimeMarker: single images, interleaved multi-image, and pure text data.
• Figure 5: Qualitative results across different scenarios. (a) multi-turn dialogue on a daily activity video, (b) temporal sentence grounding task in a beauty video, (c) event detection task in an interview video, and (d) OCR within a specific time interval.