Investigating Idiomaticity in Word Representations

TL;DR

By proposing model-agnostic measures for assessing the ability of models to capture idiomaticity, this paper contributes to determining limitations in the handling of non-compositional structures, which is one of the directions that needs to be considered for more natural, accurate and robust language understanding.

Abstract

Idiomatic expressions are an integral part of human languages, often used to express complex ideas in compressed or conventional ways (e.g. eager beaver as a keen and enthusiastic person). However, their interpretations may not be straightforwardly linked to the meanings of their individual components in isolation and this may have an impact for compositional approaches. In this paper, we investigate to what extent word representation models are able to go beyond compositional word combinations and capture multiword expression idiomaticity and some of the expected properties related to idiomatic meanings. We focus on noun compounds of varying levels of idiomaticity in two languages (English and Portuguese), presenting a dataset of minimal pairs containing human idiomaticity judgments for each noun compound at both type and token levels, their paraphrases and their occurrences in naturalistic and sense-neutral contexts, totalling 32,200 sentences. We propose this set of minimal pairs for evaluating how well a model captures idiomatic meanings, and define a set of fine-grained metrics of Affinity and Scaled Similarity, to determine how sensitive the models are to perturbations that may lead to changes in idiomaticity. The results obtained with a variety of representative and widely used models indicate that, despite superficial indications to the contrary in the form of high similarities, idiomaticity is not yet accurately represented in current models. Moreover, the performance of models with different levels of contextualisation suggests that their ability to capture context is not yet able to go beyond more superficial lexical clues provided by the words and to actually incorporate the relevant semantic clues needed for idiomaticity.

Figures (10)

• Figure 1: Distribution of cosine similarities between the minimal pairs at sentence level, with the original NC and the probe-modified substitution for English (EN, in blue) and Portuguese (PT, in orange), with naturalistic (Nat) sentences in darker shade and neutral (Neut) in lighter. The lower panel (Ideal Values) is an illustration of similarity values ideally expected for the different probes. The means and standard deviations are in Table \ref{['tab:num_sent']} in the Appendix.
• Figure 2: Distribution of cosine similarities between the minimal pairs at NC level, with the original NC and the probe-modified substitution for English (blue) and Portuguese (orange), with naturalistic sentences in darker shade and neutral in lighter. The lower panel (Ideal Values) is an illustration of similarity values ideally expected for the different probes. The means and standard deviations are in Table \ref{['tab:num_sent']} in the Appendix.
• Figure 3: Affinity at the NC level for English (blue) and Portuguese (orange), with naturalistic sentences in darker shade and neutral in lighter. The lower panel (Ideal Values) is an illustration of values ideally expected for the different affinities. The means and standard deviations are in Table \ref{['tab:num_Affinity']} in the Appendix.
• Figure 4: Affinity by idiomaticity Class at NC level for English (EN) and Portuguese (PT) naturalistic sentences. Idiomatic (I) in green, partly compositional (PC) in yellow and Compositional NCs (C) in blue.
• Figure 5: Average Scaled Similarity when the original NCs are replaced by gold synonyms (Sim$_{R|Syn}$) or by the synonyms of component words (Sim$_{R|WordsSyn}$), in relation to random substitutions. English (blue) and Portuguese (orange), with naturalistic sentences in darker shades and for neutral in lighter. The means and standard deviations are in Table \ref{['tab:num_S14_S34']} in the Appendix.