PL-MTEB: Polish Massive Text Embedding Benchmark

TL;DR

PL-MTEB addresses the evaluation gap for Polish text embeddings by introducing a 28-task benchmark across 5 task types, augmented by the Polish Library of Science Corpus (PLSC) to support clustering. It integrates with the MTEB framework and releases datasets on Hugging Face with open-source evaluation tooling. Evaluating 15 models (Polish and multilingual) across metrics such as $accuracy$, $v$-measure, $AP$, $nDCG@10$, and $Spearman$, the study finds the MMLW family generally achieves the best averages, though performance varies by task and type. This benchmark standardizes Polish embedding evaluation and is designed to benefit multilingual settings, with future expansions and ongoing leaderboard integration planned.

Abstract

In this paper, we introduce the Polish Massive Text Embedding Benchmark (PL-MTEB), a comprehensive benchmark for text embeddings in Polish. The PL-MTEB consists of 28 diverse NLP tasks from 5 task types. We adapted the tasks based on previously used datasets by the Polish NLP community. In addition, we created a new PLSC (Polish Library of Science Corpus) dataset consisting of titles and abstracts of scientific publications in Polish, which was used as the basis for two novel clustering tasks. We evaluated 15 publicly available models for text embedding, including Polish and multilingual ones, and collected detailed results for individual tasks and aggregated results for each task type and the entire benchmark. PL-MTEB comes with open-source code at https://github.com/rafalposwiata/pl-mteb.

Figures (1)

• Figure 1: An overview of tasks in PL-MTEB.