Jasper and Stella: distillation of SOTA embedding models
Dun Zhang, Jiacheng Li, Ziyang Zeng, Fulong Wang
TL;DR
To address the practicality gap of dense retrieval, the authors introduce Jasper, a 2B embedding model distilled from larger teachers through a four-stage process. They design three distillation losses (cosine, similarity, and relative similarity) and apply Matryoshka Representation Learning to compress concatenated teacher vectors into compact embeddings, with a multimodal stage for vision-language alignment. On the MTEB benchmark, Jasper achieves 71.54 on average across 56 datasets, matching several 7B-scale methods while remaining substantially smaller. The work provides public releases of the model, data, and training code, and highlights directions for efficient, cross-modal dense retrieval systems.
Abstract
A crucial component in many deep learning applications, such as Frequently Asked Questions (FAQ) and Retrieval-Augmented Generation (RAG), is dense retrieval. In this process, embedding models transform raw text into numerical vectors. However, the embedding models that currently excel on text embedding benchmarks, like the Massive Text Embedding Benchmark (MTEB), often have numerous parameters and high vector dimensionality. This poses challenges for their application in real-world scenarios. To address this issue, we propose a novel multi-stage distillation framework that enables a smaller student embedding model to distill multiple larger teacher embedding models through three carefully designed losses. Meanwhile, we utilize Matryoshka Representation Learning (MRL) to reduce the vector dimensionality of the student embedding model effectively. Our student model named Jasper with 2 billion parameters, built upon the Stella embedding model, obtained the No.3 position on the MTEB leaderboard (as of December 24, 2024), achieving an average 71.54 score across 56 datasets. We have released the model and data on the Hugging Face Hub (https://huggingface.co/infgrad/jasper_en_vision_language_v1) (https://huggingface.co/datasets/infgrad/jasper_text_distill_dataset), and the training codes are available in this project repository (https://github.com/NLPJCL/RAG-Retrieval).