Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality

Xueguang Ma, Luyu Gao, Shengyao Zhuang, Jiaqi Samantha Zhan, Jamie Callan, Jimmy Lin

TL;DR

Tevatron-v2 tackles the challenge of building retrieval systems that generalize across scale, languages, and modalities. It introduces a unified data format, memory- and inference-efficient training and serving mechanisms with vLLM integration, and Matryoshka Representation Learning to adapt embeddings at inference. The authors demonstrate a unified dense retriever capable of multilingual and multimodal retrieval, evaluated on BEIR, MIRACL, and ViDoRe, and show promising cross-modality zero-shot generalization. They also release OmniEmbed as a baseline unifying text, image, video, and audio retrieval, and provide open-source datasets and code to support reproducible research.

Abstract

Recent advancements in large language models (LLMs) have driven interest in billion-scale retrieval models with strong generalization across retrieval tasks and languages. Additionally, progress in large vision-language models has created new opportunities for multimodal retrieval. In response, we have updated the Tevatron toolkit, introducing a unified pipeline that enables researchers to explore retriever models at different scales, across multiple languages, and with various modalities. This demo paper highlights the toolkit's key features, bridging academia and industry by supporting efficient training, inference, and evaluation of neural retrievers. We showcase a unified dense retriever achieving strong multilingual and multimodal effectiveness, and conduct a cross-modality zero-shot study to demonstrate its research potential. Alongside, we release OmniEmbed, to the best of our knowledge, the first embedding model that unifies text, image document, video, and audio retrieval, serving as a baseline for future research.

Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality

TL;DR

Tevatron-v2 tackles the challenge of building retrieval systems that generalize across scale, languages, and modalities. It introduces a unified data format, memory- and inference-efficient training and serving mechanisms with vLLM integration, and Matryoshka Representation Learning to adapt embeddings at inference. The authors demonstrate a unified dense retriever capable of multilingual and multimodal retrieval, evaluated on BEIR, MIRACL, and ViDoRe, and show promising cross-modality zero-shot generalization. They also release OmniEmbed as a baseline unifying text, image, video, and audio retrieval, and provide open-source datasets and code to support reproducible research.

Abstract

Recent advancements in large language models (LLMs) have driven interest in billion-scale retrieval models with strong generalization across retrieval tasks and languages. Additionally, progress in large vision-language models has created new opportunities for multimodal retrieval. In response, we have updated the Tevatron toolkit, introducing a unified pipeline that enables researchers to explore retriever models at different scales, across multiple languages, and with various modalities. This demo paper highlights the toolkit's key features, bridging academia and industry by supporting efficient training, inference, and evaluation of neural retrievers. We showcase a unified dense retriever achieving strong multilingual and multimodal effectiveness, and conduct a cross-modality zero-shot study to demonstrate its research potential. Alongside, we release OmniEmbed, to the best of our knowledge, the first embedding model that unifies text, image document, video, and audio retrieval, serving as a baseline for future research.
Paper Structure (15 sections, 1 figure, 3 tables)

This paper contains 15 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Tevatron-v2 Overview
  3. Unified Data Management
  4. GPU Memory Efficiency
  5. Inference Efficiency
  6. Comparison with Other Toolkits
  7. Experiment: Unified Dense Retriever
  8. Setup
  9. Training data
  10. Hyperparameters
  11. Evaluation
  12. Model Variants
  13. Results
  14. Zero-Shot Generalization
  15. Conclusion

Figures (1)

  • Figure 1: Encoding speed comparison between Transformers and vLLM implementation. For text, we used a retriever based on Llama3.1-8B retriever; for Wiki-SS, we used a multimodal retriever based on QWen2-VL-2B retriever with 784×784 image inputs.