Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval

Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, Min Zhang

TL;DR

To address long-context multilingual text retrieval, the authors train a native 8k-context encoder from scratch with RoPE and unpadding, and build a hybrid text representation model (TRM) plus a cross-encoder reranker using contrastive learning. They introduce Matryoshka embeddings and a sparse representation, enabling elastic dense and sparse indexing, and optimize with a multi-task contrastive fine-tuning regime. Empirically, the TRM and reranker achieve competitive results with smaller models, particularly excelling on long-context benchmarks, while offering notable efficiency improvements for industrial deployment. The work is complemented by extensive evaluations on NLU and multilingual retrieval tasks and is released openly to foster further research and practical adoption.

Abstract

We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications.

mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval

TL;DR

To address long-context multilingual text retrieval, the authors train a native 8k-context encoder from scratch with RoPE and unpadding, and build a hybrid text representation model (TRM) plus a cross-encoder reranker using contrastive learning. They introduce Matryoshka embeddings and a sparse representation, enabling elastic dense and sparse indexing, and optimize with a multi-task contrastive fine-tuning regime. Empirically, the TRM and reranker achieve competitive results with smaller models, particularly excelling on long-context benchmarks, while offering notable efficiency improvements for industrial deployment. The work is complemented by extensive evaluations on NLU and multilingual retrieval tasks and is released openly to foster further research and practical adoption.

Abstract

We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications.
Paper Structure (58 sections, 3 equations, 5 figures, 22 tables)

This paper contains 58 sections, 3 equations, 5 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Text Encoder
  4. Unpadding Mode
  5. Data
  6. Training Curriculum
  7. Training Setup
  8. Text Representation Model
  9. Contrastive Pre-Training
  10. Matryoshka Embedding
  11. Sparse Representation
  12. Contrastive Fine-Tuning
  13. Text Reranking Model
  14. Evaluation
  15. Natural Language Understanding
  16. ...and 43 more sections

Figures (5)

  • Figure 1: Training pipeline. We first build an 8k long-context multilingual encoder. Then based on it, we train text representation and reranking models for retrieval.
  • Figure 2: Our text encoder architecture.
  • Figure 3: Our TRM and reranker.
  • Figure 4: Elastic embedding results on MTEB English.
  • Figure 5: MLDR scores in contrastive pre-training. none keeps the RoPE untouched in pre-training. 1024 and 8192 are the max sequence length in evaluations. revNTK-8912 recovers the 8k context by NTK scaling.