MELO: An Evaluation Benchmark for Multilingual Entity Linking of Occupations

Abstract

We present the Multilingual Entity Linking of Occupations (MELO) Benchmark, a new collection of 48 datasets for evaluating the linking of entity mentions in 21 languages to the ESCO Occupations multilingual taxonomy. MELO was built using high-quality, pre-existent human annotations. We conduct experiments with simple lexical models and general-purpose sentence encoders, evaluated as bi-encoders in a zero-shot setup, to establish baselines for future research. The datasets and source code for standardized evaluation are publicly available at https://github.com/Avature/melo-benchmark

Figures (7)

• Figure 1: Histogram of minimum (normalized) edit distances between each query and the closest relevant corpus element for a selection of monolingual tasks in MELO.
• Figure 2: Correlation between model performance and the median of the minimum edit distance between queries and relevant corpus elements in monolingual datasets.
• Figure 3: Top-$k$ accuracy (A@k) for a selection of models in the MELO Benchmark tasks corresponding to O*NET, Germany, Spain, the Netherlands, and Denmark.
• Figure 4: Histogram of minimum (normalized) edit distances between each query and the closest relevant corpus element for each monolingual task in MELO.
• Figure 5: Correlation between model performance and the median of the minimum edit distance between queries and relevant corpus elements in monolingual datasets.