Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings

Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, Zhicheng Dou

TL;DR

This work tackles the challenges of explicit alignment in omni-modal embeddings by introducing e5-omni, a lightweight recipe that plugs into off-the-shelf vision-language backbones. It adds three components—modality-aware temperature calibration, a controllable negative curriculum with debiasing, and batch whitening with covariance alignment—to harmonize similarity scales and geometry across diverse modalities. Empirical results on MMEB-V2 and AudioCaps show consistent improvements, with larger gains observed as model size increases and the approach transferring across backbones. The method enhances robustness of omni-modal retrieval in mixed-modality batches, offering a practical, scalable path toward unified multimodal representations.

Abstract

Modern information systems often involve different types of items, e.g., a text query, an image, a video clip, or an audio segment. This motivates omni-modal embedding models that map heterogeneous modalities into a shared space for direct comparison. However, most recent omni-modal embeddings still rely heavily on implicit alignment inherited from pretrained vision-language model (VLM) backbones. In practice, this causes three common issues: (i) similarity logits have modality-dependent sharpness, so scores are not on a consistent scale; (ii) in-batch negatives become less effective over time because mixed-modality batches create an imbalanced hardness distribution; as a result, many negatives quickly become trivial and contribute little gradient; and (iii) embeddings across modalities show mismatched first- and second-order statistics, which makes rankings less stable. To tackle these problems, we propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. e5-omni combines three simple components: (1) modality-aware temperature calibration to align similarity scales, (2) a controllable negative curriculum with debiasing to focus on confusing negatives while reducing the impact of false negatives, and (3) batch whitening with covariance regularization to better match cross-modal geometry in the shared embedding space. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines, and the same recipe also transfers well to other VLM backbones. We release our model checkpoint at https://huggingface.co/Haon-Chen/e5-omni-7B.

e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings

TL;DR

This work tackles the challenges of explicit alignment in omni-modal embeddings by introducing e5-omni, a lightweight recipe that plugs into off-the-shelf vision-language backbones. It adds three components—modality-aware temperature calibration, a controllable negative curriculum with debiasing, and batch whitening with covariance alignment—to harmonize similarity scales and geometry across diverse modalities. Empirical results on MMEB-V2 and AudioCaps show consistent improvements, with larger gains observed as model size increases and the approach transferring across backbones. The method enhances robustness of omni-modal retrieval in mixed-modality batches, offering a practical, scalable path toward unified multimodal representations.

Abstract

Modern information systems often involve different types of items, e.g., a text query, an image, a video clip, or an audio segment. This motivates omni-modal embedding models that map heterogeneous modalities into a shared space for direct comparison. However, most recent omni-modal embeddings still rely heavily on implicit alignment inherited from pretrained vision-language model (VLM) backbones. In practice, this causes three common issues: (i) similarity logits have modality-dependent sharpness, so scores are not on a consistent scale; (ii) in-batch negatives become less effective over time because mixed-modality batches create an imbalanced hardness distribution; as a result, many negatives quickly become trivial and contribute little gradient; and (iii) embeddings across modalities show mismatched first- and second-order statistics, which makes rankings less stable. To tackle these problems, we propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. e5-omni combines three simple components: (1) modality-aware temperature calibration to align similarity scales, (2) a controllable negative curriculum with debiasing to focus on confusing negatives while reducing the impact of false negatives, and (3) batch whitening with covariance regularization to better match cross-modal geometry in the shared embedding space. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines, and the same recipe also transfers well to other VLM backbones. We release our model checkpoint at https://huggingface.co/Haon-Chen/e5-omni-7B.
Paper Structure (29 sections, 7 equations, 6 figures, 7 tables)

This paper contains 29 sections, 7 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method: e5-omni
  3. Preliminaries
  4. Modality-aware Temperature Calibration
  5. Controllable Negative Curriculum
  6. Masked DCL objective.
  7. Curriculum schedule.
  8. Batch Whitening and Covariance Alignment
  9. Experiments
  10. Experimental Setup
  11. Training Datasets.
  12. Implementation Details.
  13. Evaluation.
  14. Overall Results
  15. Ablation Study
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Implicit vs. explicit alignment for omni-modal embeddings. (a) Implicit alignment leads to modality-dependent sharpness, negative hardness imbalance, and unstable ranking. (b) e5-omni performs explicit alignment with three lightweight modules to calibrate cross-modality similarities.
  • Figure 2: Overview of e5-omni. Given omni-modal inputs, e5-omni augments a VLM backbone with three lightweight components: (1) Modality-aware temperature calibration computes a modality-composition indicator $w(x)$ and applies learnable modality-specific temperatures $\boldsymbol{\tau}$ to calibrate logits $\ell(q,p)$; (2) Controllable negative curriculum progressively masks easy negatives and optimizes a masked debiased objective $\mathcal{L}_{\mathrm{DCL}}$; (3) Batch whitening and covariance alignment whitens batch embeddings and adds a CORAL-style covariance regularizer.
  • Figure 3: PCA overlap on VOC2007. Left: e5-omni w/o. alignment. Right: e5-omni. We project embeddings into a shared 2D PCA space and overlay $2\sigma$ covariance ellipses. We report centroid (distance between the query/target mean embeddings) and covgap (Frobenius gap between their covariance matrices).
  • Figure 4: Covariance-difference heatmap on VOC2007. Left: e5-omni w/o. alignment. Right: e5-omni. After a fixed 32D random projection, we visualize covdiff: the entrywise magnitude of the query--target covariance difference matrix.
  • Figure 5: The performances of e5-omni-7B under different training settings on MMEB-V2. We report the overall scores using same metric as in Table \ref{['tab:mmeb-v2-overall']}.
  • ...and 1 more figures