Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gemini Embedding: Generalizable Embeddings from Gemini

Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, Tom Duerig

TL;DR

Gemini Embedding is introduced, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model, which demonstrates strong capabilities across a broad selection of tasks and surpasses specialized domain-specific models.

Abstract

In this report, we introduce Gemini Embedding, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model. Capitalizing on Gemini's inherent multilingual and code understanding capabilities, Gemini Embedding produces highly generalizable embeddings for text spanning numerous languages and textual modalities. The representations generated by Gemini Embedding can be precomputed and applied to a variety of downstream tasks including classification, similarity, clustering, ranking, and retrieval. Evaluated on the Massive Multilingual Text Embedding Benchmark (MMTEB), which includes over one hundred tasks across 250+ languages, Gemini Embedding substantially outperforms prior state-of-the-art models, demonstrating considerable improvements in embedding quality. Achieving state-of-the-art performance across MMTEB's multilingual, English, and code benchmarks, our unified model demonstrates strong capabilities across a broad selection of tasks and surpasses specialized domain-specific models.

Gemini Embedding: Generalizable Embeddings from Gemini

TL;DR

Gemini Embedding is introduced, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model, which demonstrates strong capabilities across a broad selection of tasks and surpasses specialized domain-specific models.

Abstract

In this report, we introduce Gemini Embedding, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model. Capitalizing on Gemini's inherent multilingual and code understanding capabilities, Gemini Embedding produces highly generalizable embeddings for text spanning numerous languages and textual modalities. The representations generated by Gemini Embedding can be precomputed and applied to a variety of downstream tasks including classification, similarity, clustering, ranking, and retrieval. Evaluated on the Massive Multilingual Text Embedding Benchmark (MMTEB), which includes over one hundred tasks across 250+ languages, Gemini Embedding substantially outperforms prior state-of-the-art models, demonstrating considerable improvements in embedding quality. Achieving state-of-the-art performance across MMTEB's multilingual, English, and code benchmarks, our unified model demonstrates strong capabilities across a broad selection of tasks and surpasses specialized domain-specific models.

Paper Structure

This paper contains 38 sections, 3 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Embedding Models
  4. LLMs for Embedding Data Generation
  5. LLMs as Embedding Models
  6. Gemini Embedding
  7. Model Architecture
  8. Training Objective
  9. Recipe
  10. Pre-finetuning
  11. Finetuning
  12. Model Soup
  13. Datasets
  14. Training Data Mixture
  15. Pre-finetuning
  16. ...and 23 more sections

Figures (3)

  • Figure 1: Gemini Embedding represents text as dense vectors where semantically similar text inputs are mapped to vectors near one another in the vector space. Currently it supports more than 100+ languages, and its embeddings can be used for various tasks such as retrieval and classification.
  • Figure 2: Gemini Embedding supports cross-lingual retrieval where different languages can be used for queries and passages. We show two examples from XTREME-UP showing the strong cross-lingual retrieval capability of Gemini Embedding. Despite Assamese being a relatively low-resource language and the Hindi query having a typo, the Gemini Embedding model correctly understood the key entities and the contexts in the queries and retrieved the correct passages.
  • Figure 3: Results on retrieval datasets with different number of hard negatives. We show that our hard negatives are mostly useful.