Explaining novel senses using definition generation with open language models

TL;DR

This paper demonstrates that open-weight language models fine-tuned with instruction-tuning can generate high-quality definitions for novel word senses, outperforming the best AXOLOTL'24 Subtask 2 submissions across Finnish, Russian, and German. It compares decoder-only and encoder-decoder architectures (TowerInstruct, mT0-XL, Aya-101) and shows that both can be competitive, with performance depending on data sources and language. A key contribution is an aggregation approach using prototypical definition embeddings to produce a single sense label per target sense, along with a thorough qualitative analysis of fluency, adequacy, and circularity. The work emphasizes transparency and accessibility by releasing code and models, and highlights the importance of domain-relevant fine-tuning data for complex semantic tasks.

Abstract

We apply definition generators based on open-weights large language models to the task of creating explanations of novel senses, taking target word usages as an input. To this end, we employ the datasets from the AXOLOTL'24 shared task on explainable semantic change modeling, which features Finnish, Russian and German languages. We fine-tune and provide publicly the open-source models performing higher than the best submissions of the aforementioned shared task, which employed closed proprietary LLMs. In addition, we find that encoder-decoder definition generators perform on par with their decoder-only counterparts.