Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement

TL;DR

The paper investigates whether multilingual pretrained language representations encode abstract, cross-linguistic syntactic structures by constructing parallel BLM-Agr datasets across English, French, Italian, and Romanian and evaluating a two-level variational framework. It combines a sentence-level VAE with a task-level component to test whether chunk-based syntax can be detected within sentences and generalized across sequences, under cross-lingual and multilingual training. Across all languages, results reveal limited cross-language transfer and language-specific encoding of syntactic structure, even when overt cues exist or additional supervision is provided. This suggests that current multilingual representations rely on language-specific indicators rather than shared, abstract syntactic representations, informing future directions for improving cross-lingual syntactic generalization.

Abstract

In this paper, our goal is to investigate to what degree multilingual pretrained language models capture cross-linguistically valid abstract linguistic representations. We take the approach of developing curated synthetic data on a large scale, with specific properties, and using them to study sentence representations built using pretrained language models. We use a new multiple-choice task and datasets, Blackbird Language Matrices (BLMs), to focus on a specific grammatical structural phenomenon -- subject-verb agreement across a variety of sentence structures -- in several languages. Finding a solution to this task requires a system detecting complex linguistic patterns and paradigms in text representations. Using a two-level architecture that solves the problem in two steps -- detect syntactic objects and their properties in individual sentences, and find patterns across an input sequence of sentences -- we show that despite having been trained on multilingual texts in a consistent manner, multilingual pretrained language models have language-specific differences, and syntactic structure is not shared, even across closely related languages.

Figures (6)

• Figure 1: BLM instances for verb-subject agreement, with two attractors. The errors can be grouped in two types: (i) sequence errors: WNA= wrong nr. of attractors; WN1= wrong gram. nr. for 1$^{st}$ attractor noun (N1); WN2= wrong gram. nr. for 2$^{nd}$ attractor noun (N2); (ii) grammatical errors: AEV=agreement error on the verb; AEN1=agreement error on N1; AEN2=agreement error on N2.
• Figure 2: A two-level VAE: the sentence level learns to compress a sentence into a representation useful to solve the BLM problem on the task level.
• Figure 3: Cross-language testing for detecting chunk structure in sentence embeddings.
• Figure 4: tSNE projection of the latent representation of sentences from the training data, coloured by their chunk pattern. Different markers indicate the languages: "o" for English, "x" for French, "+" for Italian, "*" for Romanian. We note that while representations cluster by the pattern, the clusters for different languages are disjoint.
• Figure 5: Average F1 performance on training on type I data over three runs -- cross-language and multi-language