Semantics of Multiword Expressions in Transformer-Based Models: A Survey

TL;DR

The paper surveys how transformer-based models represent multiword expressions (MWEs), which exhibit variable degrees of compositionality. It analyzes intrinsic and downstream evaluations across encoder and decoder architectures, highlighting inconsistent capture and a reliance on memorization and surface cues. Key findings show that MWE meaning is often localized to early layers and enhanced by broader context and linguistic properties such as lower idiosyncrasy. The work calls for standardized, directly comparable evaluation setups and broader coverage of MWE types and languages to improve generalizability.

Abstract

Multiword expressions (MWEs) are composed of multiple words and exhibit variable degrees of compositionality. As such, their meanings are notoriously difficult to model, and it is unclear to what extent this issue affects transformer architectures. Addressing this gap, we provide the first in-depth survey of MWE processing with transformer models. We overall find that they capture MWE semantics inconsistently, as shown by reliance on surface patterns and memorized information. MWE meaning is also strongly localized, predominantly in early layers of the architecture. Representations benefit from specific linguistic properties, such as lower semantic idiosyncrasy and ambiguity of target expressions. Our findings overall question the ability of transformer models to robustly capture fine-grained semantics. Furthermore, we highlight the need for more directly comparable evaluation setups.