TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding

TL;DR

This work tackles the difficulty of reasoning over long videos with LVLMs by introducing TV-RAG, a training-free adaptor that combines temporal-aware retrieval with entropy-weighted semantic frame selection. It comprises three components: semantic entropy extraction across multimodal streams, a temporal-window BM25 retriever that aligns text cues with video timestamps, and a context-augmented reasoning pipeline with query reformulation. TV-RAG demonstrates strong, cross-model gains on Video-MME, MLVU, and LongVideoBench, outperforming several baselines and serving as a practical plug-in for existing LVLMs. The approach enhances cross-modal grounding and narrative coherence without increasing model parameters, offering a scalable upgrade path for long-video understanding in open-source and commercial settings. The work also outlines future directions for finer temporal control and broader modality integration.

Abstract

Large Video Language Models (LVLMs) have rapidly emerged as the focus of multimedia AI research. Nonetheless, when confronted with lengthy videos, these models struggle: their temporal windows are narrow, and they fail to notice fine-grained semantic shifts that unfold over extended durations. Moreover, mainstream text-based retrieval pipelines, which rely chiefly on surface-level lexical overlap, ignore the rich temporal interdependence among visual, audio, and subtitle channels. To mitigate these limitations, we propose TV-RAG, a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning. The framework contributes two main mechanisms: \emph{(i)} a time-decay retrieval module that injects explicit temporal offsets into the similarity computation, thereby ranking text queries according to their true multimedia context; and \emph{(ii)} an entropy-weighted key-frame sampler that selects evenly spaced, information-dense frames, reducing redundancy while preserving representativeness. By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning. The resulting system offers a lightweight, budget-friendly upgrade path and consistently surpasses most leading baselines across established long-video benchmarks such as Video-MME, MLVU, and LongVideoBench, confirming the effectiveness of our model. The code can be found at https://github.com/AI-Researcher-Team/TV-RAG.

Figures (5)

• Figure 1: Advantages of our TV-RAG. TV-RAG provides a temporal-aware, semantic-aware, and training-free pipeline that is easily compatible with any LVLM.
• Figure 2: The illustration of our framework TV-RAG. The pipeline begins with query decoupling, where the LVLM rewrites the user question into an explicit evidence-retrieval request, cleanly separating search from reasoning. TV-RAG then executes three tightly coupled steps: (i) a semantic-entropy selector distills high-information frames across visual, OCR, ASR and detection streams; (ii) a temporal-decay BM25 retriever ranks auxiliary texts by both lexical relevance and timestamp proximity, ensuring chronological coherence; and (iii) a bi-level reasoning routine drafts and self-verifies the answer using the retrieved evidence. By closing the loop between asking, retrieving, and reasoning without altering LVLM weights, TV-RAG simultaneously sharpens retrieval precision and elevates answer faithfulness.
• Figure 3: Performance with different sampling frames rate on Video-MME videomme when using Long-LLaVA-7B longllava as the LVLM.
• Figure 4: Qualitative result shown in Video-MME videomme benchmark when applying TV-RAG with LLaVA-Video llavavideo.
• Figure 5: Grad-CAM heatmaps of the final hidden state, alongside t-SNE projections of the user's query and keyframe features, are visualized for the example in Figure \ref{['fig_exam']}. The combined visualization makes clear that the supplementary texts retrieved by TV-RAG tighten the vision–language link: they redirect the network's attention toward the frames most pertinent to the question, thereby boosting both answer precision and contextual fidelity.

Theorems & Definitions (2)

• Remark
• Remark