Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RzenEmbed: Towards Comprehensive Multimodal Retrieval

Weijian Jian, Yajun Zhang, Dawei Liang, Chunyu Xie, Yixiao He, Dawei Leng, Yuhui Yin

TL;DR

RzenEmbed tackles universal multimodal retrieval by learning a single embedding space for text, images, videos, and visual documents using a two-stage training pipeline built on a Qwen2-VL backbone. It introduces a refined InfoNCE objective with false negative mitigation, a hardness-weighted negative mining strategy, per-task learnable temperatures, embedding prompts, and model souping to achieve robust, task-aware representations. Across MMEB-V1 and MMEB-V2 benchmarks, RzenEmbed attains state-of-the-art results, with notable gains in video and visual-document retrieval and strong cross-domain generalization. The work demonstrates practical impact for multimodal search and retrieval-augmented AI systems by delivering scalable, instruction-following embeddings with robust performance across diverse modalities.

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has extended CLIP-based frameworks to produce powerful, universal embeddings for retrieval tasks. However, existing methods primarily focus on natural images, offering limited support for other crucial visual modalities such as videos and visual documents. To bridge this gap, we introduce RzenEmbed, a unified framework to learn embeddings across a diverse set of modalities, including text, images, videos, and visual documents. We employ a novel two-stage training strategy to learn discriminative representations. The first stage focuses on foundational text and multimodal retrieval. In the second stage, we introduce an improved InfoNCE loss, incorporating two key enhancements. Firstly, a hardness-weighted mechanism guides the model to prioritize challenging samples by assigning them higher weights within each batch. Secondly, we implement an approach to mitigate the impact of false negatives and alleviate data noise. This strategy not only enhances the model's discriminative power but also improves its instruction-following capabilities. We further boost performance with learnable temperature parameter and model souping. RzenEmbed sets a new state-of-the-art on the MMEB benchmark. It not only achieves the best overall score but also outperforms all prior work on the challenging video and visual document retrieval tasks. Our models are available in https://huggingface.co/qihoo360/RzenEmbed.

RzenEmbed: Towards Comprehensive Multimodal Retrieval

TL;DR

RzenEmbed tackles universal multimodal retrieval by learning a single embedding space for text, images, videos, and visual documents using a two-stage training pipeline built on a Qwen2-VL backbone. It introduces a refined InfoNCE objective with false negative mitigation, a hardness-weighted negative mining strategy, per-task learnable temperatures, embedding prompts, and model souping to achieve robust, task-aware representations. Across MMEB-V1 and MMEB-V2 benchmarks, RzenEmbed attains state-of-the-art results, with notable gains in video and visual-document retrieval and strong cross-domain generalization. The work demonstrates practical impact for multimodal search and retrieval-augmented AI systems by delivering scalable, instruction-following embeddings with robust performance across diverse modalities.

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has extended CLIP-based frameworks to produce powerful, universal embeddings for retrieval tasks. However, existing methods primarily focus on natural images, offering limited support for other crucial visual modalities such as videos and visual documents. To bridge this gap, we introduce RzenEmbed, a unified framework to learn embeddings across a diverse set of modalities, including text, images, videos, and visual documents. We employ a novel two-stage training strategy to learn discriminative representations. The first stage focuses on foundational text and multimodal retrieval. In the second stage, we introduce an improved InfoNCE loss, incorporating two key enhancements. Firstly, a hardness-weighted mechanism guides the model to prioritize challenging samples by assigning them higher weights within each batch. Secondly, we implement an approach to mitigate the impact of false negatives and alleviate data noise. This strategy not only enhances the model's discriminative power but also improves its instruction-following capabilities. We further boost performance with learnable temperature parameter and model souping. RzenEmbed sets a new state-of-the-art on the MMEB benchmark. It not only achieves the best overall score but also outperforms all prior work on the challenging video and visual document retrieval tasks. Our models are available in https://huggingface.co/qihoo360/RzenEmbed.

Paper Structure

This paper contains 30 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Architecture
  5. Training
  6. Training Objective
  7. False Negative Mitigation
  8. Hardness-Weighted Strategy
  9. Recipe
  10. Multimodal Continual Training
  11. Fine-Tuning
  12. Merging Image Classification dataset
  13. Enhancing Video Data
  14. Task-Specific Learnable Temperature
  15. Embedding Prompt
  16. ...and 15 more sections

Figures (1)

  • Figure 1: The architecture of RzenEmbed. The model takes both visual input (images or videos) and text input (instruction + text). Visual data is processed by a vision encoder and projection layer before being fed into the LLM. The LLM jointly encodes both modalities, and the final embedding (EMB) is extracted from the last token's hidden state.