MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis

TL;DR

This work extends the Massive Text Embedding Benchmark (MTEB) to French by assembling 18 datasets across 26 tasks, including 3 newly created French datasets, and evaluating 51 embedding models. It preserves the MTEB evaluation framework and provides open-source code along with a public leaderboard to track progress as models evolve. The results show that no single model dominates all tasks, though large multilingual models optimized for sentence similarity perform strongly on average, with certain native French models excelling on specific tasks. Overall, the paper delivers a valuable French-language evaluation resource, analyzes how model characteristics relate to performance, and offers practical guidance for selecting embeddings in French NLP applications.

Abstract

Recently, numerous embedding models have been made available and widely used for various NLP tasks. The Massive Text Embedding Benchmark (MTEB) has primarily simplified the process of choosing a model that performs well for several tasks in English, but extensions to other languages remain challenging. This is why we expand MTEB to propose the first massive benchmark of sentence embeddings for French. We gather 15 existing datasets in an easy-to-use interface and create three new French datasets for a global evaluation of 8 task categories. We compare 51 carefully selected embedding models on a large scale, conduct comprehensive statistical tests, and analyze the correlation between model performance and many of their characteristics. We find out that even if no model is the best on all tasks, large multilingual models pre-trained on sentence similarity perform exceptionally well. Our work comes with open-source code, new datasets and a public leaderboard.

Figures (12)

• Figure 1: Critical difference diagram representing the significant rank gaps between models. The axis represents the normalized average rank of the models (lower is better). The black bars indicate that the difference in models' rank is not statistically significant, i.e. lower than the critical difference.
• Figure 2: Model performance depending on the language of the data they have been trained on.
• Figure 3: Cosine similarity between tasks' data. Ninety random samples per task's data are embedded using the multilingual-e5-small model. The embeddings of each task's data sample are averaged. The similarity between each dataset is then calculated using cosine similarity as in Muennighoff2022MTEBMT.
• Figure 4: 2D projection of tasks' data. 90 random samples per task's data are embedded using multlingual-e5-small model wang2022text. The embeddings are reduced to 2 dimensions using PCA. The centroid of each task's data is represented, along with the ellipse showing the standard deviation along each axis.
• Figure 5: Extracts of Syntec dataset.