Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Italian sentence embeddings properties through multi-tasking

Vivi Nastase, Giuseppe Samo, Chunyang Jiang, Paola Merlo

TL;DR

The paper investigates whether pretrained Italian sentence embeddings encode abstract linguistic information in a multi-task setting using Italian BLMs (AgrI, CausI, OdI). It introduces a two-level architecture that first compresses sentence representations and then solves BLM tasks, leveraging Electra-based embeddings and contrastive losses. Across experiments, single-task training generally outperforms multi-task training, suggesting that abstract notions like constituents or thematic roles are not consistently present in the embeddings or are not jointly capturable across tasks. The work provides curated Italian BLM datasets, analyzes error patterns across tasks, and highlights challenges in achieving cross-task sharing, pointing to directions for architectures and multilingual extensions.

Abstract

We investigate to what degree existing LLMs encode abstract linguistic information in Italian in a multi-task setting. We exploit curated synthetic data on a large scale -- several Blackbird Language Matrices (BLMs) problems in Italian -- and use them to study how sentence representations built using pre-trained language models encode specific syntactic and semantic information. We use a two-level architecture to model separately a compression of the sentence embeddings into a representation that contains relevant information for a task, and a BLM task. We then investigate whether we can obtain compressed sentence representations that encode syntactic and semantic information relevant to several BLM tasks. While we expected that the sentence structure -- in terms of sequence of phrases/chunks -- and chunk properties could be shared across tasks, performance and error analysis show that the clues for the different tasks are encoded in different manners in the sentence embeddings, suggesting that abstract linguistic notions such as constituents or thematic roles does not seem to be present in the pretrained sentence embeddings.

Exploring Italian sentence embeddings properties through multi-tasking

TL;DR

The paper investigates whether pretrained Italian sentence embeddings encode abstract linguistic information in a multi-task setting using Italian BLMs (AgrI, CausI, OdI). It introduces a two-level architecture that first compresses sentence representations and then solves BLM tasks, leveraging Electra-based embeddings and contrastive losses. Across experiments, single-task training generally outperforms multi-task training, suggesting that abstract notions like constituents or thematic roles are not consistently present in the embeddings or are not jointly capturable across tasks. The work provides curated Italian BLM datasets, analyzes error patterns across tasks, and highlights challenges in achieving cross-task sharing, pointing to directions for architectures and multilingual extensions.

Abstract

We investigate to what degree existing LLMs encode abstract linguistic information in Italian in a multi-task setting. We exploit curated synthetic data on a large scale -- several Blackbird Language Matrices (BLMs) problems in Italian -- and use them to study how sentence representations built using pre-trained language models encode specific syntactic and semantic information. We use a two-level architecture to model separately a compression of the sentence embeddings into a representation that contains relevant information for a task, and a BLM task. We then investigate whether we can obtain compressed sentence representations that encode syntactic and semantic information relevant to several BLM tasks. While we expected that the sentence structure -- in terms of sequence of phrases/chunks -- and chunk properties could be shared across tasks, performance and error analysis show that the clues for the different tasks are encoded in different manners in the sentence embeddings, suggesting that abstract linguistic notions such as constituents or thematic roles does not seem to be present in the pretrained sentence embeddings.
Paper Structure (20 sections, 9 figures, 4 tables)

This paper contains 20 sections, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The BLM task and the BLM Italian datasets
  4. BLM-AgrI -- subject-verb agreement in Italian
  5. BLM-CausI and BLM-OdI
  6. BLM-CausI context and answers
  7. BLM-OdI Context and Answers
  8. Lexicalisation
  9. Dataset statistics
  10. Multi-task representations
  11. Multi-task results
  12. Discussion
  13. Error Analysis
  14. Conclusions
  15. Appendix
  16. ...and 5 more sections

Figures (9)

  • Figure 1: BLM instances for verb-subject agreement, with two attractors. We build candidate answers displaying one of two types of errors: (i) sequence errors: WNA= wrong nr. of attractors; WN1= wrong gram. nr. for 1$^{st}$ attractor noun (N1); WN2= wrong gram. nr. for 2$^{nd}$ attractor noun (N2); (ii) grammatical errors: AEV=agreement error on the verb; AEN1=agreement error on N1; AEN2=agreement error on N2.
  • Figure 2: BLM contexts answers and their location of errors (see text) for the Change of state group (Caus) and the object drop (Od) class.
  • Figure 3: A two-level VAE: the sentence level learns to compress a sentence into a representation useful to solve the BLM problem on the task level.
  • Figure 4: Performance comparison across single-task and multi-task training paradigms for the three subtasks (single task darker shade of each colour, multi-task lighter shade), trained on type-I data, tested on the three types, and averaged over three independent runs. Results obtained using the Italian Electra pretrained model.
  • Figure 5: Error analysis for agreement: multi- vs. single task, training on type I data, testing on all.
  • ...and 4 more figures