Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages

TL;DR

The paper investigates whether monolingual BERTs encode linguistic structure aligned with typological similarities by comparing layer-wise weight matrices across languages using biCKA, grounded in WALS-derived language vectors. It demonstrates that syntactic information tends to emerge in middle layers ($4$–$6$) and is reflected in weight-matrix similarities, with domain adaptation on a parallel corpus strengthening these signals. The study introduces biCKA to capture bidirectional covariance in weight matrices and shows that attention-related matrices, particularly $V$ and $OA$, carry salient syntactic information. These findings offer a path toward actionable explainability and guided cross-language training strategies, while acknowledging limitations from pretraining data diversity and incomplete typological resources.

Abstract

The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language. In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices.

Figures (7)

• Figure 1: Spearman's correlation coefficient over all language pairs ranked with typological features and with weight matrix similarities. Rows are the matrices types, and columns are the layers. Values closer to $+1$ are in red, closer to 0 are in black, and values closer to $-1$ are in blue. Statistically significant results with a $p-value<0.01$ are labeled with $^*$.
• Figure 2: Each matrix shows Spearman’s correlation coefficients for extra-cluster typological similarities and weight matrix similarities. Rows are the matrices types, and columns are the layers. Values closer to $+1$ are in red, closer to 0 are in black, and values closer to -1 are in blue. Statistically significant results with a $p-value<0.01$ are labeled with $^*$.
• Figure 3: Spearman's correlation coefficient over European language pairs ranked with typological features and with weight matrix of domain adapted BERT similarities \ref{['fig:finetuning/syn_post']} and of pretrained BERT similarities. Rows are the matrices types, and columns are the layers. Values closer to $+1$ are in red, closer to 0 are in black, and values closer to -1 are in blue. Statistically significant results with a $p-value<0.01$ are labeled with $^*$.
• Figure 4: t-SNE plot of clustering based on syntactic (\ref{['synclusterin']}) and morphological (\ref{['morphclustering']}) features extracted from WALS.
• Figure 5: Each matrix shows the Spearman's correlation coefficients for extra-cluster morphological analysis, one matrix for each pair of clusters, $M_i$ and $M_j$. Values closer to $+1$ are in red, values closer to -1 in blue. Statistically significant results with a p-value lower than $0.01$ are labelled with $^*$.