Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis

TL;DR

This study advances the interpretability of contextualized word representations by focusing on Spanish lexical ambiguity in both monolingual and multilingual BERT-based models. It introduces SAW-C, a carefully controlled Spanish minimal-pair sentence dataset with human relatedness norms, and uses it to probe how well pre-trained LMs capture sense distinctions and sense-graded relatedness. Across BETO and a broad set of Spanish models, results show that LM embeddings partially track human judgments but fall short of inter-annotator agreement, with distinct layer-wise trajectories that depend on model architecture and scale. The work highlights the value and limitations of current Spanish LM representations for disambiguation tasks and points to scaling and architectural factors as key determinants of their sensitivity to sense boundaries, with implications for multilingual NLP and linguistic resource development.

Abstract

Lexical ambiguity -- where a single wordform takes on distinct, context-dependent meanings -- serves as a useful tool to compare across different language models' (LMs') ability to form distinct, contextualized representations of the same stimulus. Few studies have systematically compared LMs' contextualized word embeddings for languages beyond English. Here, we evaluate semantic representations of Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models. We develop a novel dataset of minimal-pair sentences evoking the same or different sense for a target ambiguous noun. In a pre-registered study, we collect contextualized human relatedness judgments for each sentence pair. We find that various BERT-based LMs' contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark. In exploratory work, we find that performance scales with model size. We also identify stereotyped trajectories of target noun disambiguation as a proportion of traversal through a given LM family's architecture, which we partially replicate in English. We contribute (1) a dataset of controlled, Spanish sentence stimuli with human relatedness norms, and (2) to our evolving understanding of the impact that LM specification (architectures, training protocols) exerts on contextualized embeddings.

Figures (12)

• Figure 1: Sample item from task. Participants emitted graded relatedness judgments from a scale of 1 (totally unrelated) to 5 (same sense) for a given target word (here: aceite -- oil), using information provided by the context cue across the sentence pair (here: de oliva / de motor -- olive / motor).
• Figure 2: Density plot representing the distribution of mean relatedness judgments for sentence pairs. As expected, word meanings were rated as more related when used in Same Sense than Different Sense contexts.
• Figure 3: Average Cosine Distance between the contextualized representations of the target ambiguous word across each layer of BETO, depicted as a function of whether the contexts cued the Same Sense or Different Sense.
• Figure 4: Distribution of human inter-annotator agreement scores, calculated using a leave-one-annotator out scheme. The vertical dashed line represents the correlation between human judgments and Cosine Distance values extracted from BETO.
• Figure 5: Residuals of linear regression models fit for each LM, predicting relatedness from the interaction between cosine distance and layer position; residual distributions are separable as a function of Sense Relationship.