Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AXOLOTL'24 Shared Task on Multilingual Explainable Semantic Change Modeling

Mariia Fedorova, Timothee Mickus, Niko Partanen, Janine Siewert, Elena Spaziani, Andrey Kutuzov

TL;DR

AXOLOTL'24 introduces the first multilingual framework for explainable semantic change modeling, proposing two tasks: identifying novel senses and producing dictionary-like definitions for those senses across Finnish, Russian, and a German surprise language. The organizers construct sense-annotated diachronic datasets from public lexicographic resources and formalize evaluation protocols that couple WSD/WSI with definition modeling, using metrics such as ARI, macro-F1, and BERTScore alongside BLEU. Results from six teams show that masked-language-model–based approaches are robust across languages, while generative LMs can be competitive, underscoring cross-lingual transfer and the value of lexicographic data for explainability. The work yields publicly available datasets and baselines, advancing the intersection of NLP and historical linguistics and highlighting future directions for improving sense inventories and cross-language definitional generation."

Abstract

This paper describes the organization and findings of AXOLOTL'24, the first multilingual explainable semantic change modeling shared task. We present new sense-annotated diachronic semantic change datasets for Finnish and Russian which were employed in the shared task, along with a surprise test-only German dataset borrowed from an existing source. The setup of AXOLOTL'24 is new to the semantic change modeling field, and involves subtasks of identifying unknown (novel) senses and providing dictionary-like definitions to these senses. The methods of the winning teams are described and compared, thus paving a path towards explainability in computational approaches to historical change of meaning.

AXOLOTL'24 Shared Task on Multilingual Explainable Semantic Change Modeling

TL;DR

AXOLOTL'24 introduces the first multilingual framework for explainable semantic change modeling, proposing two tasks: identifying novel senses and producing dictionary-like definitions for those senses across Finnish, Russian, and a German surprise language. The organizers construct sense-annotated diachronic datasets from public lexicographic resources and formalize evaluation protocols that couple WSD/WSI with definition modeling, using metrics such as ARI, macro-F1, and BERTScore alongside BLEU. Results from six teams show that masked-language-model–based approaches are robust across languages, while generative LMs can be competitive, underscoring cross-lingual transfer and the value of lexicographic data for explainability. The work yields publicly available datasets and baselines, advancing the intersection of NLP and historical linguistics and highlighting future directions for improving sense inventories and cross-language definitional generation."

Abstract

This paper describes the organization and findings of AXOLOTL'24, the first multilingual explainable semantic change modeling shared task. We present new sense-annotated diachronic semantic change datasets for Finnish and Russian which were employed in the shared task, along with a surprise test-only German dataset borrowed from an existing source. The setup of AXOLOTL'24 is new to the semantic change modeling field, and involves subtasks of identifying unknown (novel) senses and providing dictionary-like definitions to these senses. The methods of the winning teams are described and compared, thus paving a path towards explainability in computational approaches to historical change of meaning.
Paper Structure (36 sections, 7 figures, 9 tables, 1 algorithm)

This paper contains 36 sections, 7 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Prior work and state of the field
  3. Data
  4. Finnish
  5. Data sources.
  6. Data annotation.
  7. Russian
  8. Data sources.
  9. Data annotation.
  10. Data post-processing.
  11. German
  12. Data source.
  13. AXOLOTL'24 organization
  14. Subtask 1. Bridging diachronic word uses and a synchronic dictionary
  15. Evaluation.
  16. ...and 21 more sections

Figures (7)

  • Figure 1: Distribution of the number of unique senses per target word in the AXOLOTL'24 datasets. Cases with more than 25 senses clipped.
  • Figure 2: Distribution of the number of examples per target word.
  • Figure 3: Distribution of number of unique senses per target word in the Russian gold test data, the winner team's prediction and the best predictions for Russian by ARI (the WooperNLP team).
  • Figure 4: Number of unique senses per word in Russian, the gold test data.
  • Figure 5: Number of unique senses per word in Russian, the Deep-change team.
  • ...and 2 more figures