T2VIndexer: A Generative Video Indexer for Efficient Text-Video Retrieval

TL;DR

Text–video retrieval is often hampered by linear-time candidate evaluation over large video sets. This work introduces T2VIndexer, a sequence-to-sequence generative video indexer that directly generates video identifiers through a semantic Vi-SemTree, producing fast, constant-time pre-selection of candidates. It couples a multi-view query expansion pipeline using a Multimodal Large Language Model with a Prefix-Aware Weight-Adaptor decoder to map queries to SemIDs, and employs a two-stage retrieval: generative pre-selection followed by a conventional high-accuracy re-ranker on a small candidate set. The approach achieves substantial efficiency gains (30–50% faster) while maintaining or improving recall across MSR-VTT, MSVD, DiDeMo, and ActivityNet, and scales to large datasets. With easy integration into existing pipelines and strong empirical results, it offers a practical, scalable solution for real-time multimodal search.

Abstract

Current text-video retrieval methods mainly rely on cross-modal matching between queries and videos to calculate their similarity scores, which are then sorted to obtain retrieval results. This method considers the matching between each candidate video and the query, but it incurs a significant time cost and will increase notably with the increase of candidates. Generative models are common in natural language processing and computer vision, and have been successfully applied in document retrieval, but their application in multimodal retrieval remains unexplored. To enhance retrieval efficiency, in this paper, we introduce a model-based video indexer named T2VIndexer, which is a sequence-to-sequence generative model directly generating video identifiers and retrieving candidate videos with constant time complexity. T2VIndexer aims to reduce retrieval time while maintaining high accuracy. To achieve this goal, we propose video identifier encoding and query-identifier augmentation approaches to represent videos as short sequences while preserving their semantic information. Our method consistently enhances the retrieval efficiency of current state-of-the-art models on four standard datasets. It enables baselines with only 30\%-50\% of the original retrieval time to achieve better retrieval performance on MSR-VTT (+1.0%), MSVD (+1.8%), ActivityNet (+1.5%), and DiDeMo (+0.2%). The code is available at https://github.com/Lilidamowang/T2VIndexer-generativeSearch.

Figures (5)

• Figure 1: (a) Two stream method with independent video and text encoders. (b) Video sparse sampling for efficiency boost. (c) Our generative video indexer for efficiency boost.
• Figure 2: An overview of our T2VIndexer. T2VIndexer uses two different strategies for the training and inference stages. For the training stage, as shown in (a), the process of dividing the training set into a tree structure is illustrated. Training stage (b) shows the process of achieving multi-view query expansion through MLLM. (c) presents the pipeline for model training. For the inference stage, the new video is first inserted into the semantic tree and assigned a SemID, and the baseline model provides the precise retrieval results.
• Figure 3: Overview of the generative model of T2VIndexer. Decoder uses different parameters when generating tokens at different positions.
• Figure 4: Visualization of the difference between Generative Results and Ground-truth. We show the top-3 Generated SemIDs for each text query. The truly matched results are marked in green boxes and the falsely matched results are in red boxes.
• Figure 5: Different truncation length and top k SemID for T2VIndexer on MSR-VTT-1kA.