Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Well Do Large Language Models Disambiguate Swedish Words?

Richard Johansson

TL;DR

A battery of recent large language models on two benchmarks for word sense disambiguation in Swedish are evaluated, with a focus on how to express the set of possible senses in a given context.

Abstract

We evaluate a battery of recent large language models on two benchmarks for word sense disambiguation in Swedish. At present, all current models are less accurate than the best supervised disambiguators in cases where a training set is available, but most models outperform graph-based unsupervised systems. Different prompting approaches are compared, with a focus on how to express the set of possible senses in a given context. The best accuracies are achieved when human-written definitions of the senses are included in the prompts.

How Well Do Large Language Models Disambiguate Swedish Words?

TL;DR

A battery of recent large language models on two benchmarks for word sense disambiguation in Swedish are evaluated, with a focus on how to express the set of possible senses in a given context.

Abstract

We evaluate a battery of recent large language models on two benchmarks for word sense disambiguation in Swedish. At present, all current models are less accurate than the best supervised disambiguators in cases where a training set is available, but most models outperform graph-based unsupervised systems. Different prompting approaches are compared, with a focus on how to express the set of possible senses in a given context. The best accuracies are achieved when human-written definitions of the senses are included in the prompts.

Paper Structure

This paper contains 21 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. The SALDO lexicon
  3. Method
  4. Datasets and Experimental Setup
  5. SENSEVAL-2
  6. Eukalyptus
  7. Preprocessing
  8. Experimental setup
  9. Baselines
  10. Included Models
  11. Prompt Design
  12. Experiments
  13. Neighborhoods or Definitions?
  14. Model-written Sense Definitions
  15. Conclusions
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Fragments of SALDO neighborhoods for two of the senses of ämne. Primary descriptor edges are drawn as solid arrows and secondary descriptor edges as dashed arrows.