Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax

Iuliia Zaitova, Vitalii Hirak, Badr M. Abdullah, Dietrich Klakow, Bernd Möbius, Tania Avgustinova

TL;DR

The paper investigates how fine-tuned BERT-based encoders allocate attention to two MWE types—idioms and microsyntactic units—across six Indo-European languages. By analyzing layer-wise attention in pre-trained and task-tuned models on syntactic (DepRel, POS) and semantic (NER, Topic) tasks, it reveals distinct patterns: semantic fine-tuning yields a more even cross-layer attention for idioms, while syntactic fine-tuning increases MSU attention in lower layers. The study contributes a multilingual dataset and trained models, showing cross-language and cross-task variations in attention that reflect linguistic properties of MWEs. These findings advance understanding of how transformer attention adapts to complex lexical and syntactic phenomena, with implications for multilingual MWE processing and evaluation.

Abstract

This study analyzes the attention patterns of fine-tuned encoder-only models based on the BERT architecture (BERT-based models) towards two distinct types of Multiword Expressions (MWEs): idioms and microsyntactic units (MSUs). Idioms present challenges in semantic non-compositionality, whereas MSUs demonstrate unconventional syntactic behavior that does not conform to standard grammatical categorizations. We aim to understand whether fine-tuning BERT-based models on specific tasks influences their attention to MWEs, and how this attention differs between semantic and syntactic tasks. We examine attention scores to MWEs in both pre-trained and fine-tuned BERT-based models. We utilize monolingual models and datasets in six Indo-European languages - English, German, Dutch, Polish, Russian, and Ukrainian. Our results show that fine-tuning significantly influences how models allocate attention to MWEs. Specifically, models fine-tuned on semantic tasks tend to distribute attention to idiomatic expressions more evenly across layers. Models fine-tuned on syntactic tasks show an increase in attention to MSUs in the lower layers, corresponding with syntactic processing requirements.

Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax

TL;DR

The paper investigates how fine-tuned BERT-based encoders allocate attention to two MWE types—idioms and microsyntactic units—across six Indo-European languages. By analyzing layer-wise attention in pre-trained and task-tuned models on syntactic (DepRel, POS) and semantic (NER, Topic) tasks, it reveals distinct patterns: semantic fine-tuning yields a more even cross-layer attention for idioms, while syntactic fine-tuning increases MSU attention in lower layers. The study contributes a multilingual dataset and trained models, showing cross-language and cross-task variations in attention that reflect linguistic properties of MWEs. These findings advance understanding of how transformer attention adapts to complex lexical and syntactic phenomena, with implications for multilingual MWE processing and evaluation.

Abstract

This study analyzes the attention patterns of fine-tuned encoder-only models based on the BERT architecture (BERT-based models) towards two distinct types of Multiword Expressions (MWEs): idioms and microsyntactic units (MSUs). Idioms present challenges in semantic non-compositionality, whereas MSUs demonstrate unconventional syntactic behavior that does not conform to standard grammatical categorizations. We aim to understand whether fine-tuning BERT-based models on specific tasks influences their attention to MWEs, and how this attention differs between semantic and syntactic tasks. We examine attention scores to MWEs in both pre-trained and fine-tuned BERT-based models. We utilize monolingual models and datasets in six Indo-European languages - English, German, Dutch, Polish, Russian, and Ukrainian. Our results show that fine-tuning significantly influences how models allocate attention to MWEs. Specifically, models fine-tuned on semantic tasks tend to distribute attention to idiomatic expressions more evenly across layers. Models fine-tuned on syntactic tasks show an increase in attention to MSUs in the lower layers, corresponding with syntactic processing requirements.
Paper Structure (23 sections, 3 figures, 3 tables)

This paper contains 23 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Datasets
  5. Microsyntactic Unit Dataset
  6. Idiom Dataset
  7. English, German, Dutch, and Polish Idiom Dataset.
  8. Russian Idiom Dataset.
  9. Ukrainian Idiom Dataset.
  10. Models
  11. Fine-tuning Process
  12. Syntactic Tasks
  13. Semantic Tasks
  14. Training and Evaluation
  15. Experimental Setup
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Layer-wise attention distribution in BERT-based models for idioms and microsyntactic units (MSUs). Results shown for English (EN) and Ukrainian (UK). Models fine-tuned on syntactic tasks (Dependency Relation Classification -- DepRel, Part-of-Speech Tagging -- POS) are on the left, and semantic tasks (Named Entity Recognition -- NER, Topic Classification -- Topic) are on the right. The y-axis shows the percentage of attention scores directed from other sentence tokens towards Multiword Expressions, with higher percentages indicating stronger attention focus.
  • Figure 2: Layer-wise attention distribution within Multiword Expressions (MWEs) in German (DE), Dutch (NL), and Russian (RU) BERT-based models. The graphs show attention distribution within tokens of idioms and microsyntactic units (MSUs), comparing pre-trained models with those fine-tuned on syntactic tasks (Dependency Relation Classification -- DepRel, Part-of-Speech Tagging -- POS) and semantic tasks (Named Entity Recognition -- NER, Topic Classification -- Topic). The y-axis represents the percentage of attention between tokens within the same MWE.
  • Figure 3: Differences in Attention Patterns for Idioms and Microsyntactic Units (MSUs) in Russian. The bars show layer-wise changes in attention percentage after fine-tuning on syntactic tasks (Dependency Relation Classification – DepRel, Part-of-Speech Tagging – POS) and semantic tasks (Named Entity Recognition – NER, Topic Classification – Topic). Positive values indicate increased attention and negative values show decreased attention compared to the pre-trained model.